Forum Discussion
VS Code Synapse Extension Kernel unable to load Spark Lighter
I am trying to use the vs code extension to connect to my fabric workspace. I am able to see my workspace, download and open a notebook but when I try to execute any spark commands I get this error in return:
12:54:42,170 root INFO Should attach session in dev mode False
12:54:42,756 root INFO Starting session 8c486d38-3681-4afa-8ce7-f97646a90a6c...
12:54:42,756 msal.authority INFO Initializing with Entra authority: https://login.windows.net/common
12:54:43,509 root INFO Getting refresh token...
12:54:45,22 root INFO <session_management.SessionStatus object at 0x0000020D3FB2E560>
12:54:45,22 root ERROR Failed to initialize Spark Lighter variables. Create session failed {'code': 'InternalError', 'subCode': 0, 'message': 'An internal error occurred.', 'timeStamp': '2024-07-12T18:54:44.78251Z', 'httpStatusCode': 500, 'hresult': -2147467259, 'details': [{'code': 'RootActivityId', 'message': '69054038-af73-477d-a144-16a758fba666'}, {'code': 'Param1', 'message': 'Response status code does not indicate success: 401 (Unauthorized).'}]}
12:54:45,22 root INFO Registering Spark Lighter magics for IPython...
12:54:45,22 root INFO Registered Spark Lighter magics for IPython.
3 Replies
HI minh_nguyen ,
Accounting to the error message, it seems like related to 'spark lighter' initialize failed, have you check if this installed and configured correctly?
BTW, does these spark commands invoke the functions in spark Utility? As the document mentions, current it seems not support when you work on the desktop mode:
VS Code extension overview - Microsoft Fabric | Microsoft Learn
Regards,
Xiaoxin Sheng
I am installing it through the extension and I have tried uninstalling and reinstalling the extension.
I am trying to use a spark.read command to load a file in the lakehouse.HI minh_nguyen,
As the document mentions, current the 'Microsoft Spark Utilities' not support to use on desktop mode. Have you check if your script and functions required this library?
Introduction to Microsoft Spark utilities - Azure Synapse Analytics | Microsoft Learn
Regards,
Xiaoxin Sheng
