spark not defined using VSCode Synapse Extension

[Tom117]

I am attempting to work on a pyspark notebook locally using VS Code and the Microsoft Synapse extension.

I installed all dependencies as described in the documentation for the extension. I've connected to a workspace and opened my notebook. However, when attempting to execute my first block I am seeing that "spark" is not defined. I've attempted all runtime interpreters with no success:

I even attempted to install pyspark manually in the terminal so I could create a new session, but then pyspark is also unknown. I've been doing allot of searching around trying to figure out how all this is supposed to work but with no luck. Just calling out in the dark here, wondering if anyone might be able to assist. 

Thank you.

[SamuelHakansson]

I got the same problem when installing using the wrong version of Java, and after installing the recommended Java version and reinstalling VS code (also removed folders with extensions and settings to be sure), it worked

[Anonymous]

Hi @Tom117 

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet.
In case if you have any resolution please do share that with the community as it can be helpful to others.
Otherwise, will respond back with the more details and we will try to help.

Best Regards,
Jing

[Anonymous]

Hi  @Tom117 

After some time I finally got it work at my end. Below are what I have done before it works. Not sure which step is the key but you may have a try!

1. Use Synapse VS Code extension v1.9.0. This version fixes an issue related to the default lakehouse of notebook, which occurs in v1.8.0. 

 

2. Delete the notebook from your local drive. Delete the notebook folder in your Local Work Folder Path.

 

3. Sign off your account from Synapse VS Code extension. 

4. On Jupyter extension settings page, extend Jupyter Interrupt Timeout and Jupyter Launch Timeout. I set 100000 and 600000 for them. 

5. Close VS Code and restart it. Sign in acount to Synapse VS Code extension.

6. Download the notebook to local drive again. This will re-generate a configure file for the notebook. 

7. Open the notebook and run a code cell. Although it shows "spark is not defined", it run successfully as a result. 

 

Hope this would be helpful. 

Best Regards,
Jing
If this post helps, please Accept it as Solution to help other members find it. Appreciate your Kudos!

[Tom117]

Thank you so much for the suggestion here. I really appreciate your time. However, I gave that solution a try with no success 😕

[Anonymous]

Hi @Tom117 

I’m sorry to hear that the solution I provided did not bring a success. I understand how frustrating this can be. Let's further analyze this issue. 

1. Check whether the configuration information of the notebook is generated correctly. You can find a configuration file via path like <your Local Work Folder path>\<workspace id>\SynapseNotebook\<notebook id>\conf\lighter-config.json. It's in the following format, ensure the information is correct. Note: Do not share the file content because it includes sensitive data.

2. Check whether the session started successfully. You can find two log files in the path <your Local Work Folder path>\<workspace id>\logs\<notebook id>. 

Open the log files to check if there is any error occurring. If a session started successfully, you will see something like below.

PySparkLighter.log

SparkLight.log

If the session didn't start successfully, did it return any error message? Can you share the error message if possible? Please remove sensitive data.

Best Regards,
Jing

[Tom117]

Hey, thank you very much for getting back to me!

So, my config looks just about the same with the exception of the "auto_session" and "session_id" properties. They do not exist in my config. 

Additionally, I looked for the log files, however they also do not exist. 

Any idea why that might be the case? Perhaps my instillation has never attempted to connect to the remote? 

[Anonymous]

Hi @Tom117 

It seems the remote session has not been triggered or started successfully. Can you check the versions of the prerequisite installations, especially the version of the jdk? From this thread, it appears openjdk21 is not supported and openjdk8 is required. And ensure the jdk bin and conda bin paths are added to the Path environment variable on your system. You could add them to the Path environment variable for both user variables and system variables at the same time. 

Then restart your computer, open VS Code, delete the notebook from local and from the Local Work Folder. Download a notebook again, open it and wait a while for the kernel to be ready. After that, try running a code cell to start the remote session. 

Best Regards,
Jing

[Anonymous]

Hi @Tom117 

I met the same error and couldn't run any cell in my notebook in VS Code either. I've seen some old posts about this error and will need some time to research and test if there was a workaround that may work. If there is any progress, I will update here.

Best Regards,
Jing