VS Code notebook Spark failure

[BiJoe]

In my workspace I have two notebooks, where one is actively edited in VS Code Synapse extension, and mostly works. The only hickup is that all Spark methods are called out with "spark is not defined" Pylance message.

The other notebook is newly created and tested OK with some basic Pyspark commands in Fabric online. However, when I try to run a cell in VS Code, for example

spark.conf.set("spark.sql.parquet.vorder.enabled", "true")
spark.conf.set("spark.microsoft.delta.optimizeWrite.enabled", "true")
spark.conf.set("spark.microsoft.delta.optimizeWrite.binSize", "1073741824")

I get

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[1], line 1
----> 1 spark.conf.set("spark.sql.parquet.vorder.enabled", "true")
      2 spark.conf.set("spark.microsoft.delta.optimizeWrite.enabled", "true")
      3 spark.conf.set("spark.microsoft.delta.optimizeWrite.binSize", "1073741824")

AttributeError: 'NoneType' object has no attribute 'conf'

 For both notebooks, the Fabric Synapse runtimes load seemingly correctly, and I've tried both 1.1 and 1.2:

base                     C:\ProgramData\anaconda3
fabric-synapse-runtime-1-1     C:\ProgramData\anaconda3\envs\fabric-synapse-runtime-1-1
fabric-synapse-runtime-1-2     C:\ProgramData\anaconda3\envs\fabric-synapse-runtime-1-2


Check if synapse spark kernel env has created... true
The synapse spark 3.4 kernel env has been created, skip the creation process of conda env.
The lighter Lib already exists and the version meets requirement, skip download.
Initialization of synapse spark 3.4 kernel env has finished.
-----------------------------------------------------------
The Synapse PySpark environment preparation is complete! Now, please switch your kernel to start exploring and working with it!
===========================================================

Screenshot of how the two notebooks are showing in my VS Code, note two different icons:

 Any clues to what might cause this problem?

[MMBI]

Hi @BiJoe , I am also facing the same issue as yours, when i try to run my fabric code through VS.

My kernel is -> fabric-synapse-runtime-1-2 

And when i run any spark related commands, im getting the same error -> AttributeError: 'NoneType' object has no attribute 'sql'

I am able to run any basic print statements, but not anything related to 'spark'.

As per your solution, I tried to uninstall VS Code and reinstall the extensions but unfortunately it didnt work and I am stuck at the same place. 

Any kind of help/solution would be really appreciated.

Thanks.

[BiJoe]

@MMBI hi, sorry I abandoned using the Spark notebooks a while ago, so I don't know if this issue is resolved.

[MMBI]

@BiJoe No problem, thanks for the reply!

[BiJoe · Accepted solution]

I completely uninstalled VS Code, and reinstalled extensions. Now the notebook works as it should

[BiJoe]

@Anonymous thanks 🙂 I have Synapse Spark kernel 3.4 installed, and have tried both runtimes 1.1 and 1.2

[Anonymous]

Hi @BiJoe 
Thanks for using Fabric Community.
Apologies for the issue you have been facing. May I know which kernel are you using when this issue occured?

[Anonymous]

Hi @BiJoe 
Can you please provide the log for this error? Just copy paste the error log here.
Thanks.

[Anonymous]

Hi @BiJoe 
We haven’t heard from you on the last response and was just checking back to see if you provide the details asked above. Please let us know if you have any further queries.
Thanks