Forum Discussion
Synapse Extension unable to execute notebooks
I was initially under the impression that any notebook that works in workspace will run without trouble in vs studio code locally as I am using the fabric-synapse runtime but that is not the case.
What am I doing wrong?
One more
However, as per this 8.20 -9.07 spark commands should execute as desired.
7 Replies
Hi smpa01 , In Fabric or Synapse, the Spark session is automatically created/managed when a cluster is ready for notebook to use. However, if you're running this locally (e.g. VS Code), you need to initiate it manually with SparkSession.builder.getOrCreate().
from pyspark.sql import SparkSessionspark = SparkSession.builder.appName("Test").getOrCreate()
digitalbrain thanks for looking into it, but it did not work
Synapse:Check Extension set up shows conda 24.7.1 Checking if conda is installed ... true User Agent is node/20.15.1 (windows_nt 10.0.19045; win32; x64) # conda environments: # C:\Users\user1\AppData\Local\anaconda3 base C:\Users\user1\AppData\Local\miniconda3 fabric-synapse-runtime-1-1 C:\Users\user1\AppData\Local\miniconda3\envs\fabric-synapse-runtime-1-1 fabric-synapse-runtime-1-2 C:\Users\user1\AppData\Local\miniconda3\envs\fabric-synapse-runtime-1-2 c:\Users\user1\AppData\Local\miniconda3 Check if the old synapse spark kernel env has created... false The old synapse spark kernel env has not been created, skip the step. C:\Users\user1\AppData\Local\miniconda3 C:\Users\user1 Upgrade the existing lighter scripts to the latest versions. The file c:/users/user1/.ipython/profile_default/startup/init_lighter.py already exists and the version meets requirement, skip download. Upgrade the existing lighter scripts to the latest versions. The file c:/users/user1/.ipython/profile_default/startup/session_management.py already exists and the version meets requirement, skip download. The file C:\Users\user1\AppData\Local\miniconda3\envs\fabric-synapse-runtime-1-1\lib\site-packages\delta.pth already exists, skip download. # conda environments: # C:\Users\user1\AppData\Local\anaconda3 base C:\Users\user1\AppData\Local\miniconda3 fabric-synapse-runtime-1-1 C:\Users\user1\AppData\Local\miniconda3\envs\fabric-synapse-runtime-1-1 fabric-synapse-runtime-1-2 C:\Users\user1\AppData\Local\miniconda3\envs\fabric-synapse-runtime-1-2 c:\Users\user1\AppData\Local\miniconda3 Check if synapse spark kernel env has created... true The synapse spark 3.3 kernel env has been created, skip the creation process of conda env. The lighter Lib already exists and the version meets requirement, skip download. Initialization of synapse spark 3.3 kernel env has finished. ----------------------------------------------------------- C:\Users\user1\AppData\Local\miniconda3 C:\Users\user1 Upgrade the existing lighter scripts to the latest versions. The file c:/users/user1/.ipython/profile_default/startup/init_lighter.py already exists and the version meets requirement, skip download. Upgrade the existing lighter scripts to the latest versions. The file c:/users/user1/.ipython/profile_default/startup/session_management.py already exists and the version meets requirement, skip download. Delete the old delta.pth file... true The file C:\Users\user1\AppData\Local\miniconda3\envs\fabric-synapse-runtime-1-2\lib\site-packages\delta_spark34.pth already exists, skip download. # conda environments: # C:\Users\user1\AppData\Local\anaconda3 base C:\Users\user1\AppData\Local\miniconda3 fabric-synapse-runtime-1-1 C:\Users\user1\AppData\Local\miniconda3\envs\fabric-synapse-runtime-1-1 fabric-synapse-runtime-1-2 C:\Users\user1\AppData\Local\miniconda3\envs\fabric-synapse-runtime-1-2 c:\Users\user1\AppData\Local\miniconda3 Check if synapse spark kernel env has created... true The synapse spark 3.4 kernel env has been created, skip the creation process of conda env. The lighter Lib already exists and the version meets requirement, skip download. Initialization of synapse spark 3.4 kernel env has finished. ----------------------------------------------------------- The Synapse PySpark environment preparation is complete! Now, please switch your kernel to start exploring and working with it! ===========================================================and it runs for eternity to fetch a single row table
PySparkLighter.Log shows
14:25:55,633 urllib3.connectionpool WARNING Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1007)'))': /common/discovery/instance?authorization_endpoint=https%3A%2F%2Flogin.windows.net%2Fcommon%2Foauth2%2Fv2.0%2Fauthorize&api-version=1.0 14:25:55,722 root ERROR Failed to initialize Spark Lighter variables. HTTPSConnectionPool(host='login.microsoftonline.com', port=443): Max retries exceeded with url: /common/discovery/instance?authorization_endpoint=https%3A%2F%2Flogin.windows.net%2Fcommon%2Foauth2%2Fv2.0%2Fauthorize&api-version=1.0 (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1007)'))) 14:25:55,722 root INFO Registering Spark Lighter magics for IPython... 14:25:55,723 root INFO Registered Spark Lighter magics for IPython.Hi smpa01 , I just tried VS code in browser and it works fine there. May I know what extension did you use in your local VS code to connect with workspace?
