Trident-Spark-IDE authentication with each spark job defintion run

[Broeks]

In addition to my other topic (SparkContext not initializing when using notebooks... - Microsoft Fabric Community), where I'm trying to get notebooks in VSCode to work, I'm also trying to get spark job definitions to work in VSCode.

I've set-up VSCode based on the following youtube video and documentation:
https://www.youtube.com/watch?v=A9SjAyZ_JSc
https://learn.microsoft.com/en-us/fabric/data-engineering/setup-vs-code-extension
https://learn.microsoft.com/en-us/fabric/data-engineering/author-sjd-with-vs-code

I'm running the following code as test:

import sys
import os
#import Constant
from pyspark.sql import SparkSession
from pyspark.conf import SparkConf


if __name__ == "__main__":

  #Spark session builder
  conf = SparkConf()
  conf.set("spark.lighter.client.plugin", "org.apache.spark.lighter.DefaultLighterClientPlugin")
  conf.set("spark.sql.catalogImplementation", "lighter")
  conf.set("spark.lighter.sessionState.implementation", "org.apache.spark.sql.lighter.client.SparkLighterSessionStateBuilder")
  conf.set("spark.lighter.externalCatalog.implementation", "org.apache.spark.sql.lighter.client.ConnectCatalogClient")
  conf.set("spark.driver.host", "localhost")
  spark_session =  SparkSession.builder.config(conf=conf).getOrCreate()

  spark_context = spark_session.sparkContext
  spark_context.setLogLevel("DEBUG")
  
  #tableName = "yellowtripdata"
  # You can download the sample CSV file from this site "https://www.nyc.gov/site/tlc/about/tlc-trip-record-data.page" and upload it to the files section of the lakehouse. 
  csvFilePath = "Files/Data/dimension_customer.csv"
  #deltaTablePath = SaveToLH + "/Tables/" + tableName
  deltaTablePath = "Tables/yellowtrip"

  df = spark_session.read.format('csv').options(header='true', inferschema='true').load(csvFilePath)
  df.write.mode('overwrite').format('delta').save(deltaTablePath)

And I've though the code works and I'm able to run pyspark code locally there are some issues:

- In addition to the provided steps I've had to add the following to my config:

conf.set("spark.driver.host", "localhost")

But the real issue is, that I have to authenticate to the Trident-Spark-IDE everytime i want to run my code.

I've noticed that I needed to authenticate through the debug logging, where is says:

[ForkJoinPool.commonPool-worker-25] AADTokenClient: To sign in, use a web browser to open the page https://microsoft.com/devicelogin and enter the code XXXXXXXX to authenticate.

After I've authenticated the code runs, but I'm not able to authenticate only once. I'm signed into my account through the Synapse VS Code extension, so that can't be the issue. How can I ensure that I remain logged in?

[Broeks · Accepted solution]

The issues seemed to be releated to the version of the extension that I was using. It's fixed now, so upgrading the extension to 1.9.1. did the trick.

[Anonymous]

Hi @Broeks ,

The reason you need to authenticate more than once may be because the token for authentication expires.

You can configure the token lifetime, as described below:

Configure tokens - Azure Active Directory B2C | Microsoft Learn

If you have any other questions please feel free to contact me.

Best Regards,
Yang
Community Support Team

If there is any post helps, then please consider Accept it as the solution  to help the other members find it more quickly.
If I misunderstand your needs or you still have problems on it, please feel free to let us know. Thanks a lot!