topic Re: VS Code notebook Spark failure in Data Engineering

VS Code notebook Spark failure

BiJoe — Thu, 01 Feb 2024 09:33:30 GMT

In my workspace I have two notebooks, where one is actively edited in VS Code Synapse extension, and mostly works. The only hickup is that all Spark methods are called out with "spark is not defined" Pylance message.

The other notebook is newly created and tested OK with some basic Pyspark commands in Fabric online. However, when I try to run a cell in VS Code, for example

spark.conf.set("spark.sql.parquet.vorder.enabled", "true") spark.conf.set("spark.microsoft.delta.optimizeWrite.enabled", "true") spark.conf.set("spark.microsoft.delta.optimizeWrite.binSize", "1073741824")

I get

--------------------------------------------------------------------------- AttributeError Traceback (most recent call last) Cell In[1], line 1 ----> 1 spark.conf.set("spark.sql.parquet.vorder.enabled", "true") 2 spark.conf.set("spark.microsoft.delta.optimizeWrite.enabled", "true") 3 spark.conf.set("spark.microsoft.delta.optimizeWrite.binSize", "1073741824") AttributeError: 'NoneType' object has no attribute 'conf'

For both notebooks, the Fabric Synapse runtimes load seemingly correctly, and I've tried both 1.1 and 1.2:

base C:\ProgramData\anaconda3 fabric-synapse-runtime-1-1 C:\ProgramData\anaconda3\envs\fabric-synapse-runtime-1-1 fabric-synapse-runtime-1-2 C:\ProgramData\anaconda3\envs\fabric-synapse-runtime-1-2 Check if synapse spark kernel env has created... true The synapse spark 3.4 kernel env has been created, skip the creation process of conda env. The lighter Lib already exists and the version meets requirement, skip download. Initialization of synapse spark 3.4 kernel env has finished. ----------------------------------------------------------- The Synapse PySpark environment preparation is complete! Now, please switch your kernel to start exploring and working with it! ===========================================================

Screenshot of how the two notebooks are showing in my VS Code, note two different icons:

Any clues to what might cause this problem?

Re: VS Code notebook Spark failure

Anonymous — Fri, 02 Feb 2024 04:17:33 GMT

Hi @BiJoe
Thanks for using Fabric Community.
Apologies for the issue you have been facing. May I know which kernel are you using when this issue occured?

Re: VS Code notebook Spark failure

BiJoe — Fri, 02 Feb 2024 08:07:28 GMT

@Anonymous thanks 🙂 I have Synapse Spark kernel 3.4 installed, and have tried both runtimes 1.1 and 1.2

Re: VS Code notebook Spark failure

Anonymous — Fri, 02 Feb 2024 11:51:21 GMT

Hi @BiJoe
Can you please provide the log for this error? Just copy paste the error log here.
Thanks.

Re: VS Code notebook Spark failure

Anonymous — Mon, 05 Feb 2024 05:32:50 GMT

Hi @BiJoe
We haven’t heard from you on the last response and was just checking back to see if you provide the details asked above. Please let us know if you have any further queries.
Thanks

Re: VS Code notebook Spark failure

BiJoe — Mon, 05 Feb 2024 13:48:28 GMT

I completely uninstalled VS Code, and reinstalled extensions. Now the notebook works as it should

Re: VS Code notebook Spark failure

Anonymous — Tue, 28 May 2024 13:26:54 GMT

Hi @BiJoe , I am also facing the same issue as yours, when i try to run my fabric code through VS.

My kernel is -> fabric-synapse-runtime-1-2

And when i run any spark related commands, im getting the same error -> AttributeError: 'NoneType' object has no attribute 'sql'

I am able to run any basic print statements, but not anything related to 'spark'.

As per your solution, I tried to uninstall VS Code and reinstall the extensions but unfortunately it didnt work and I am stuck at the same place.

Any kind of help/solution would be really appreciated.

Thanks.

Re: VS Code notebook Spark failure

BiJoe — Wed, 29 May 2024 08:02:09 GMT

@Anonymous hi, sorry I abandoned using the Spark notebooks a while ago, so I don't know if this issue is resolved.

Re: VS Code notebook Spark failure

Anonymous — Wed, 29 May 2024 08:18:15 GMT

@BiJoe No problem, thanks for the reply!