topic Re: VS Code notebook failure - Spark in Data Engineering

VS Code notebook failure - Spark

Anonymous — Tue, 28 May 2024 14:57:43 GMT

I want to integrate VS code with my Fabric workspace and I have been able to successfully do it. I have followed all the pre-requisites mentioned in this link - https://learn.microsoft.com/en-us/fabric/data-engineering/setup-vs-code-extension.

But when I try to run any spark related commands in VS code, I get an error - "AttributeError: 'NoneType' object has no attribute 'sql'" and "spark is not defined Pylance". However these commands are successfully running in Fabric.

My kernel specification is - fabric-synapse-runtime-1-2(Python 3.10.12)

As a quick fix, I tried to reinstall VS Code and the extension but unfortunately nothing worked.
Would appreciate any quick help or solution on this.

Thanks.

Re: VS Code notebook failure - Spark

Anonymous — Wed, 29 May 2024 05:44:18 GMT

Hi @Anonymous

Thanks for using Fabric Community.
Please follow these steps:
1) Right click the ipython notebook

Reveal in File Explorer

2) Open the python log file: ../../../logs/${NOTEBOOK_ARTIFACT_ID}/PySparkLighter.log and check the session failure reason

Please attach the screenshot of the error here to understand what is happening.

You can also try stopping the spark session in the Notebook and try running in VS code.
Thanks

Re: VS Code notebook failure - Spark

Anonymous — Wed, 29 May 2024 07:43:30 GMT

Hi @Anonymous,

Here is the screenshot of the logs that i can see.

Also how do I stop the Spark session in notebook and run in VS code?

Thanks.

Re: VS Code notebook failure - Spark

Anonymous — Mon, 03 Jun 2024 09:41:57 GMT

The solution that worked for me is - Re-installing VS Code by clearing the cache files as well during uninstallation.

Re: VS Code notebook failure - Spark

Anonymous — Mon, 03 Jun 2024 09:47:31 GMT

Hi @Anonymous
Glad that your query got resolved. Please continue using Fabric Community for any help regarding your queries.