Forum Discussion

Broeks's avatar
Broeks
Helper II
2 years ago
Solved

SparkContext not initializing when using notebooks in VSCode

Hello, I've been working on getting the spark job definition running locally. And although I've come along way, I'm not quite there yet.    I've set-up VSCode based on the following youtube video ...
Data Integration
  • Anonymous's avatar
    Anonymous
    2 years ago

    Hi Broeks,

    AFAIK, you still need to import the libraries even if they already exist on your local machine. In my opinion, I'd like to suggest you import used libraries on the notebook before execute the codes.
    When you use the online version notebook with default Lakehouse, they will auto initialize some variables/configuration and libraries so that you can direct use them without import. (they did not works when you works on the local environment)

    In addition, I'd like to suggest you check the document limitation part if they meet to your scenario: 

    VS Code extension overview - Microsoft Fabric | Microsoft Learn

    The extension under the desktop mode doesn't support the Microsoft Spark Utilities yet

    Regards,

    Xiaoxin Sheng

Broeks's avatar
Broeks
Helper II
2 years ago

I've started again from scratch and a thing that i noticed is that I need to install a spark version manually. Seems obvious, but the manual doesn't mention it, so this might be a good addition to the manual (VS Code extension overview - Microsoft Fabric | Microsoft Learn). So i've installed version 3.4.1 of spark to match with Runtime 1.2 (Apache Spark runtime in Fabric - Microsoft Fabric | Microsoft Learn)

 

Still the same error though.
I'm running on a trail with a capacity i don't hav access to:


Can this be releated to the errors i'm having. I'm able to run notebooks in the browser, but not in VSCode