Forum Discussion
SparkContext not initializing when using notebooks in VSCode
Hi Broeks,
AFAIK, you still need to import the libraries even if they already exist on your local machine. In my opinion, I'd like to suggest you import used libraries on the notebook before execute the codes.
When you use the online version notebook with default Lakehouse, they will auto initialize some variables/configuration and libraries so that you can direct use them without import. (they did not works when you works on the local environment)In addition, I'd like to suggest you check the document limitation part if they meet to your scenario:
VS Code extension overview - Microsoft Fabric | Microsoft Learn
The extension under the desktop mode doesn't support the Microsoft Spark Utilities yet
Regards,
Xiaoxin Sheng
I've started again from scratch and a thing that i noticed is that I need to install a spark version manually. Seems obvious, but the manual doesn't mention it, so this might be a good addition to the manual (VS Code extension overview - Microsoft Fabric | Microsoft Learn). So i've installed version 3.4.1 of spark to match with Runtime 1.2 (Apache Spark runtime in Fabric - Microsoft Fabric | Microsoft Learn)
Still the same error though.
I'm running on a trail with a capacity i don't hav access to:
Can this be releated to the errors i'm having. I'm able to run notebooks in the browser, but not in VSCode
Hi Broeks,
AFAIK, you still need to import the libraries even if they already exist on your local machine. In my opinion, I'd like to suggest you import used libraries on the notebook before execute the codes.
When you use the online version notebook with default Lakehouse, they will auto initialize some variables/configuration and libraries so that you can direct use them without import. (they did not works when you works on the local environment)
In addition, I'd like to suggest you check the document limitation part if they meet to your scenario:
VS Code extension overview - Microsoft Fabric | Microsoft Learn
The extension under the desktop mode doesn't support the Microsoft Spark Utilities yet
Regards,
Xiaoxin Sheng
Thanks for your suggestion. I've got it running!
The things i've done to make it work:
- Run VSCode as admin: this enable VScode to install te required packages while building the environment
- Create a spark context with the config provided for the spark job definitions.
- Added the code:conf.set("spark.driver.host", "localhost")This resulted in the following code to create te spark context:
from pyspark.sql import SparkSession from pyspark.conf import SparkConf #Spark session builder conf = SparkConf() conf.set("spark.lighter.client.plugin", "org.apache.spark.lighter.DefaultLighterClientPlugin") conf.set("spark.sql.catalogImplementation", "lighter") conf.set("spark.lighter.sessionState.implementation", "org.apache.spark.sql.lighter.client.SparkLighterSessionStateBuilder") conf.set("spark.lighter.externalCatalog.implementation", "org.apache.spark.sql.lighter.client.ConnectCatalogClient") conf.set("spark.driver.host", "localhost") spark = SparkSession.builder.config(conf=conf).getOrCreate()My suggestion would be to add a code snippet to the built-in folder, similair to the spark job defintions.
Thanks for the support!