Synapse Extension unable to execute notebooks

[smpa01]

I was initially under the impression that any notebook that works in workspace will run without trouble in vs studio code locally  as I am using the fabric-synapse runtime but that is not the case.

What am I doing wrong?

One more

However, as per this 8.20 -9.07 spark commands should execute as desired.

[digitalbrain]

Hi @smpa01 , In Fabric or Synapse, the Spark session is automatically created/managed when a cluster is ready for notebook to use. However, if you're running this locally (e.g. VS Code), you need to initiate it manually with SparkSession.builder.getOrCreate().

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("Test").getOrCreate()

[smpa01]

@digitalbrain  thanks for looking into it, but it did not work

Synapse:Check Extension set up shows


conda 24.7.1

Checking if conda is installed ... true
User Agent is node/20.15.1 (windows_nt 10.0.19045; win32; x64)
# conda environments:
#
                         C:\Users\user1\AppData\Local\anaconda3
base                     C:\Users\user1\AppData\Local\miniconda3
fabric-synapse-runtime-1-1     C:\Users\user1\AppData\Local\miniconda3\envs\fabric-synapse-runtime-1-1
fabric-synapse-runtime-1-2     C:\Users\user1\AppData\Local\miniconda3\envs\fabric-synapse-runtime-1-2
                         c:\Users\user1\AppData\Local\miniconda3


Check if the old synapse spark kernel env has created... false
The old synapse spark kernel env has not been created, skip the step.
C:\Users\user1\AppData\Local\miniconda3

C:\Users\user1

Upgrade the existing lighter scripts to the latest versions.
The file c:/users/user1/.ipython/profile_default/startup/init_lighter.py already exists and the version meets requirement, skip download.
Upgrade the existing lighter scripts to the latest versions.
The file c:/users/user1/.ipython/profile_default/startup/session_management.py already exists and the version meets requirement, skip download.
The file C:\Users\user1\AppData\Local\miniconda3\envs\fabric-synapse-runtime-1-1\lib\site-packages\delta.pth already exists, skip download.
# conda environments:
#
                         C:\Users\user1\AppData\Local\anaconda3
base                     C:\Users\user1\AppData\Local\miniconda3
fabric-synapse-runtime-1-1     C:\Users\user1\AppData\Local\miniconda3\envs\fabric-synapse-runtime-1-1
fabric-synapse-runtime-1-2     C:\Users\user1\AppData\Local\miniconda3\envs\fabric-synapse-runtime-1-2
                         c:\Users\user1\AppData\Local\miniconda3


Check if synapse spark kernel env has created... true
The synapse spark 3.3 kernel env has been created, skip the creation process of conda env.
The lighter Lib already exists and the version meets requirement, skip download.
Initialization of synapse spark 3.3 kernel env has finished.
-----------------------------------------------------------
C:\Users\user1\AppData\Local\miniconda3

C:\Users\user1

Upgrade the existing lighter scripts to the latest versions.
The file c:/users/user1/.ipython/profile_default/startup/init_lighter.py already exists and the version meets requirement, skip download.
Upgrade the existing lighter scripts to the latest versions.
The file c:/users/user1/.ipython/profile_default/startup/session_management.py already exists and the version meets requirement, skip download.
Delete the old delta.pth file... true
The file C:\Users\user1\AppData\Local\miniconda3\envs\fabric-synapse-runtime-1-2\lib\site-packages\delta_spark34.pth already exists, skip download.
# conda environments:
#
                         C:\Users\user1\AppData\Local\anaconda3
base                     C:\Users\user1\AppData\Local\miniconda3
fabric-synapse-runtime-1-1     C:\Users\user1\AppData\Local\miniconda3\envs\fabric-synapse-runtime-1-1
fabric-synapse-runtime-1-2     C:\Users\user1\AppData\Local\miniconda3\envs\fabric-synapse-runtime-1-2
                         c:\Users\user1\AppData\Local\miniconda3


Check if synapse spark kernel env has created... true
The synapse spark 3.4 kernel env has been created, skip the creation process of conda env.
The lighter Lib already exists and the version meets requirement, skip download.
Initialization of synapse spark 3.4 kernel env has finished.
-----------------------------------------------------------
The Synapse PySpark environment preparation is complete! Now, please switch your kernel to start exploring and working with it!
===========================================================

and it runs for eternity to fetch a single row table

PySparkLighter.Log shows

14:25:55,633 urllib3.connectionpool WARNING Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1007)'))': /common/discovery/instance?authorization_endpoint=https%3A%2F%2Flogin.windows.net%2Fcommon%2Foauth2%2Fv2.0%2Fauthorize&api-version=1.0
14:25:55,722 root ERROR Failed to initialize Spark Lighter variables. HTTPSConnectionPool(host='login.microsoftonline.com', port=443): Max retries exceeded with url: /common/discovery/instance?authorization_endpoint=https%3A%2F%2Flogin.windows.net%2Fcommon%2Foauth2%2Fv2.0%2Fauthorize&api-version=1.0 (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1007)')))
14:25:55,722 root INFO Registering Spark Lighter magics for IPython...
14:25:55,723 root INFO Registered Spark Lighter magics for IPython.

[smpa01]

Can someone please help me with that

@DennesTorres @frithjof_v @AndyDDC 

[digitalbrain]

Hi @smpa01 , I just tried VS code in browser and it works fine there. May I know what extension did you use in your local VS code to connect with workspace?

[smpa01]

I am not running in vs studio code web (it works fine there), I am running it in vs studio code in pc.

[digitalbrain]

Thank you for the confirmation. Do you use any extension to connect Fabric worspace (to fetch the notebook) in your VS code? I'm trying to see how you set "fabric-synapse runtime" in your VS code to replicate it at my end.

[smpa01]

It was painful enough (still does not work as stated above), following are the combin of resources  i have used, in order of preferences

1. https://www.red-gate.com/simple-talk/blogs/microsoft-fabric-using-vs-code-to-develop-notebooks/

3. https://youtu.be/0tODoHLXhMc?si=m8sGA6SiTVk0m3sC

2.https://youtu.be/iS2qaWLv_g8?si=mRq8tkQYtQVRozRb (same author as 1)

4. https://youtu.be/A9SjAyZ_JSc?si=kJJ-tMEEY6WZwycG

Documentation - https://learn.microsoft.com/en-us/fabric/data-engineering/setup-vs-code-extension