Synapse Extension unable to execute notebooks

Question

I was initially under the impression that any notebook that works in workspace will run without trouble in vs studio code locally&nbsp; as I am using the fabric-synapse runtime but that is not the case.
&nbsp;
What am I doing wrong?
&nbsp;

&nbsp;
One more
&nbsp;

&nbsp;

&nbsp;
However, as per this 8.20 -9.07 spark commands should execute as desired.

digitalbrain · Answer

Hi&nbsp;smpa01&nbsp;, In Fabric or Synapse, the Spark session is automatically created/managed when a cluster is ready for notebook to use. However, if you're running this locally (e.g. VS Code), you need to initiate it manually with SparkSession.builder.getOrCreate().from pyspark.sql import SparkSessionspark = SparkSession.builder.appName("Test").getOrCreate()

smpa01 · Answer

&nbsp;
digitalbrain&nbsp; thanks for looking into it, but it did not work
&nbsp;
Synapse:Check Extension set up shows

conda 24.7.1

Checking if conda is installed ... true
User Agent is node/20.15.1 (windows_nt 10.0.19045; win32; x64)
# conda environments:
#
                         C:\Users\user1\AppData\Local\anaconda3
base                     C:\Users\user1\AppData\Local\miniconda3
fabric-synapse-runtime-1-1     C:\Users\user1\AppData\Local\miniconda3\envs\fabric-synapse-runtime-1-1
fabric-synapse-runtime-1-2     C:\Users\user1\AppData\Local\miniconda3\envs\fabric-synapse-runtime-1-2
                         c:\Users\user1\AppData\Local\miniconda3

Check if the old synapse spark kernel env has created... false
The old synapse spark kernel env has not been created, skip the step.
C:\Users\user1\AppData\Local\miniconda3

C:\Users\user1

Upgrade the existing lighter scripts to the latest versions.
The file c:/users/user1/.ipython/profile_default/startup/init_lighter.py already exists and the version meets requirement, skip download.
Upgrade the existing lighter scripts to the latest versions.
The file c:/users/user1/.ipython/profile_default/startup/session_management.py already exists and the version meets requirement, skip download.
The file C:\Users\user1\AppData\Local\miniconda3\envs\fabric-synapse-runtime-1-1\lib\site-packages\delta.pth already exists, skip download.
# conda environments:
#
                         C:\Users\user1\AppData\Local\anaconda3
base                     C:\Users\user1\AppData\Local\miniconda3
fabric-synapse-runtime-1-1     C:\Users\user1\AppData\Local\miniconda3\envs\fabric-synapse-runtime-1-1
fabric-synapse-runtime-1-2     C:\Users\user1\AppData\Local\miniconda3\envs\fabric-synapse-runtime-1-2
                         c:\Users\user1\AppData\Local\miniconda3

Check if synapse spark kernel env has created... true
The synapse spark 3.3 kernel env has been created, skip the creation process of conda env.
The lighter Lib already exists and the version meets requirement, skip download.
Initialization of synapse spark 3.3 kernel env has finished.
-----------------------------------------------------------
C:\Users\user1\AppData\Local\miniconda3

C:\Users\user1

Upgrade the existing lighter scripts to the latest versions.
The file c:/users/user1/.ipython/profile_default/startup/init_lighter.py already exists and the version meets requirement, skip download.
Upgrade the existing lighter scripts to the latest versions.
The file c:/users/user1/.ipython/profile_default/startup/session_management.py already exists and the version meets requirement, skip download.
Delete the old delta.pth file... true
The file C:\Users\user1\AppData\Local\miniconda3\envs\fabric-synapse-runtime-1-2\lib\site-packages\delta_spark34.pth already exists, skip download.
# conda environments:
#
                         C:\Users\user1\AppData\Local\anaconda3
base                     C:\Users\user1\AppData\Local\miniconda3
fabric-synapse-runtime-1-1     C:\Users\user1\AppData\Local\miniconda3\envs\fabric-synapse-runtime-1-1
fabric-synapse-runtime-1-2     C:\Users\user1\AppData\Local\miniconda3\envs\fabric-synapse-runtime-1-2
                         c:\Users\user1\AppData\Local\miniconda3

Check if synapse spark kernel env has created... true
The synapse spark 3.4 kernel env has been created, skip the creation process of conda env.
The lighter Lib already exists and the version meets requirement, skip download.
Initialization of synapse spark 3.4 kernel env has finished.
-----------------------------------------------------------
The Synapse PySpark environment preparation is complete! Now, please switch your kernel to start exploring and working with it!
===========================================================

&nbsp;
and it runs for eternity to fetch a single row table

&nbsp;
PySparkLighter.Log shows
14:25:55,633 urllib3.connectionpool WARNING Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1007)'))': /common/discovery/instance?authorization_endpoint=https%3A%2F%2Flogin.windows.net%2Fcommon%2Foauth2%2Fv2.0%2Fauthorize&amp;api-version=1.0
14:25:55,722 root ERROR Failed to initialize Spark Lighter variables. HTTPSConnectionPool(host='login.microsoftonline.com', port=443): Max retries exceeded with url: /common/discovery/instance?authorization_endpoint=https%3A%2F%2Flogin.windows.net%2Fcommon%2Foauth2%2Fv2.0%2Fauthorize&amp;api-version=1.0 (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1007)')))
14:25:55,722 root INFO Registering Spark Lighter magics for IPython...
14:25:55,723 root INFO Registered Spark Lighter magics for IPython.

digitalbrain · Answer

Hi&nbsp;smpa01&nbsp;, I just tried VS code in browser and it works fine there. May I know what extension did you use in your local VS code to connect with workspace?&nbsp;

smpa01 · Answer

I am not running in vs studio code web (it works fine there), I am running it in vs studio code in pc.

digitalbrain · Answer

Thank you for the confirmation. Do you use any extension to connect Fabric worspace (to fetch the notebook) in your VS code? I'm trying to see how you set "fabric-synapse runtime" in your VS code to replicate it at my end.

Forum Discussion

Related content

Can't start kernel from notebook in VS Code synapse remote extension

Issue with Initializing Synapse Spark Environment Using Synapse VS Code Extension

Notebook to query Synapse Serverless SQL

Write to Fabric OneLake from a Synapse Spark notebook

Synapse Notebook UI thinks cell is running. Notebook won't proceed.