Forum Discussion
SparkContext not initializing when using notebooks in VSCode
Hi Broeks,
AFAIK, you still need to import the libraries even if they already exist on your local machine. In my opinion, I'd like to suggest you import used libraries on the notebook before execute the codes.
When you use the online version notebook with default Lakehouse, they will auto initialize some variables/configuration and libraries so that you can direct use them without import. (they did not works when you works on the local environment)In addition, I'd like to suggest you check the document limitation part if they meet to your scenario:
VS Code extension overview - Microsoft Fabric | Microsoft Learn
The extension under the desktop mode doesn't support the Microsoft Spark Utilities yet
Regards,
Xiaoxin Sheng
Hi Broeks,
AFAIK, you still need to import the libraries even if they already exist on your local machine. In my opinion, I'd like to suggest you import used libraries on the notebook before execute the codes.
When you use the online version notebook with default Lakehouse, they will auto initialize some variables/configuration and libraries so that you can direct use them without import. (they did not works when you works on the local environment)
In addition, I'd like to suggest you check the document limitation part if they meet to your scenario:
VS Code extension overview - Microsoft Fabric | Microsoft Learn
The extension under the desktop mode doesn't support the Microsoft Spark Utilities yet
Regards,
Xiaoxin Sheng
Thanks for your suggestion. I've got it running!
The things i've done to make it work:
- Run VSCode as admin: this enable VScode to install te required packages while building the environment
- Create a spark context with the config provided for the spark job definitions.
- Added the code:
conf.set("spark.driver.host", "localhost")This resulted in the following code to create te spark context:
from pyspark.sql import SparkSession
from pyspark.conf import SparkConf
#Spark session builder
conf = SparkConf()
conf.set("spark.lighter.client.plugin", "org.apache.spark.lighter.DefaultLighterClientPlugin")
conf.set("spark.sql.catalogImplementation", "lighter")
conf.set("spark.lighter.sessionState.implementation", "org.apache.spark.sql.lighter.client.SparkLighterSessionStateBuilder")
conf.set("spark.lighter.externalCatalog.implementation", "org.apache.spark.sql.lighter.client.ConnectCatalogClient")
conf.set("spark.driver.host", "localhost")
spark = SparkSession.builder.config(conf=conf).getOrCreate()My suggestion would be to add a code snippet to the built-in folder, similair to the spark job defintions.
Thanks for the support!