Forum Discussion

Tom117's avatar
Tom117
Frequent Visitor
2 years ago

spark not defined using VSCode Synapse Extension

I am attempting to work on a pyspark notebook locally using VS Code and the Microsoft Synapse extension.   I installed all dependencies as described in the documentation for the extension. I've con...
workspaces
Anonymous's avatar
Anonymous
Not applicable
2 years ago

Hi  Tom117 

 

After some time I finally got it work at my end. Below are what I have done before it works. Not sure which step is the key but you may have a try!

 

1. Use Synapse VS Code extension v1.9.0. This version fixes an issue related to the default lakehouse of notebook, which occurs in v1.8.0. 

 

 

2. Delete the notebook from your local drive. Delete the notebook folder in your Local Work Folder Path.

 

 

3. Sign off your account from Synapse VS Code extension. 

 

4. On Jupyter extension settings page, extend Jupyter Interrupt Timeout and Jupyter Launch Timeout. I set 100000 and 600000 for them. 

 

5. Close VS Code and restart it. Sign in acount to Synapse VS Code extension.

 

6. Download the notebook to local drive again. This will re-generate a configure file for the notebook. 

 

7. Open the notebook and run a code cell. Although it shows "spark is not defined", it run successfully as a result. 

 

Hope this would be helpful. 

 

Best Regards,
Jing
If this post helps, please Accept it as Solution to help other members find it. Appreciate your Kudos!

 

 

 

 

 

  • Tom117's avatar
    Tom117
    Frequent Visitor
    2 years ago

    Thank you so much for the suggestion here. I really appreciate your time. However, I gave that solution a try with no success 😕

    • Anonymous's avatar
      Anonymous
      Not applicable
      2 years ago

      Hi Tom117 

       

      I’m sorry to hear that the solution I provided did not bring a success. I understand how frustrating this can be. Let's further analyze this issue. 

       

      1. Check whether the configuration information of the notebook is generated correctly. You can find a configuration file via path like <your Local Work Folder path>\<workspace id>\SynapseNotebook\<notebook id>\conf\lighter-config.json. It's in the following format, ensure the information is correct. Note: Do not share the file content because it includes sensitive data.

       

      2. Check whether the session started successfully. You can find two log files in the path <your Local Work Folder path>\<workspace id>\logs\<notebook id>. 

      Open the log files to check if there is any error occurring. If a session started successfully, you will see something like below.

      PySparkLighter.log

      SparkLight.log

       

      If the session didn't start successfully, did it return any error message? Can you share the error message if possible? Please remove sensitive data.

       

      Best Regards,
      Jing

      • Tom117's avatar
        Tom117
        Frequent Visitor
        2 years ago

        Hey, thank you very much for getting back to me!

        So, my config looks just about the same with the exception of the "auto_session" and "session_id" properties. They do not exist in my config. 

        Additionally, I looked for the log files, however they also do not exist. 

        Any idea why that might be the case? Perhaps my instillation has never attempted to connect to the remote?