Forum Discussion

paatrickd's avatar
paatrickd
New Member
2 years ago
Solved

VS Code Synapse integration not working

Hi,

 

I want to open a notebook from Fabric in Visual Studio Code.

 

I followed the installation guide and managed to activate the Synapse Kernal and open a notebook in VS Code, however it does not recognize spark. See the code below, spark is of Nonetype.

 

 

What should I do?

data engineering
data science
data warehouse
  • Anonymous's avatar
    Anonymous
    2 years ago

    Hi paatrickd 
    Thanks for using Fabric Community.
    Can you please follow the below troubleshooting steps and provide the details:

    1) Right click the ipython notebook.

     




    2) Open the python log file: ../../logs/${NOTEBOOK_ARTIFACT_ID}/PySparkLighter.log and check the session failure reason.


    Please provide the error details so that I can help you better.

    Thank you.

  • Anonymous's avatar
    Anonymous
    2 years ago

    Hi paatrickd 
    Thanks for the update.
    Maybe you have installed conda, but forgot to set the environment variable. Please check that.
    After creating the conda environment, it will install the spark-lighter-lib whl package from blob. The whl package will be automatically upgraded when the version in the conda environment does not match the minimum required version of the extension. If there is any issue with the upgration, you could run the below code :

    conda activate ${CONDA_ENV_NAME}
pip uninstall spark-lighter-lib
pip install https://path/to/spark-lighter-lib.whl


    Also try these steps.

    1. Open the VS Code command palette and select/type 'Python: Create Terminal'
    2. In the terminal, run the following command: conda install pyspark
    3. Restart VS Code and try again.


    Hope this helps. Please let me know if you have any further questions.

  • Anonymous's avatar
    Anonymous
    2 years ago

    Hi paatrickd ,
    Go to the upper folder to find a folder called logs.

     

13 Replies

  • Anonymous's avatar
    Anonymous
    Not applicable
    2 years ago

    Hi paatrickd 
    Thanks for using Fabric Community.
    Can you please follow the below troubleshooting steps and provide the details:

    1) Right click the ipython notebook.

     




    2) Open the python log file: ../../logs/${NOTEBOOK_ARTIFACT_ID}/PySparkLighter.log and check the session failure reason.


    Please provide the error details so that I can help you better.

    Thank you.

    • Anonymous's avatar
      Anonymous
      Not applicable
      2 years ago

      Hi paatrickd ,
      Glad that your query got resolved. Please continue using Fabric Community for any help regarding your queries.

  • paatrickd's avatar
    paatrickd
    New Member
    2 years ago

    Hi,

     

    Thank you for your response.

     

    However, I do not have any logs. When following your troubleshooting steps the directory does not contain any log folder.

     

     

     

    • Anonymous's avatar
      Anonymous
      Not applicable
      2 years ago

      Hi paatrickd 
      Thanks for the update.
      Maybe you have installed conda, but forgot to set the environment variable. Please check that.
      After creating the conda environment, it will install the spark-lighter-lib whl package from blob. The whl package will be automatically upgraded when the version in the conda environment does not match the minimum required version of the extension. If there is any issue with the upgration, you could run the below code :

      conda activate ${CONDA_ENV_NAME}
pip uninstall spark-lighter-lib
pip install https://path/to/spark-lighter-lib.whl


      Also try these steps.

      1. Open the VS Code command palette and select/type 'Python: Create Terminal'
      2. In the terminal, run the following command: conda install pyspark
      3. Restart VS Code and try again.


      Hope this helps. Please let me know if you have any further questions.

      • BiJoe's avatar
        BiJoe
        Helper II
        2 years ago

        This answer solved my problem not being able to install runtime Apache Spark runtime 1.2, which is installed automatically after runtime 1.1. The Conda environment had no pyspark library due to a wheel download error. In PowerShell I activated the 1.2 runtime Conda environment, then 

         

        pip install https://tridentvscodeextension.blob.core.windows.net/spark-lighter-lib/spark34/spark_lighter_lib-34.0.0.3-py3-none-any.whl

         

        took approximately 10 minutes.  After the download completed, Synapse Pyspark was installed successfully.

    • Anonymous's avatar
      Anonymous
      Not applicable
      2 years ago

      Hi paatrickd ,
      Go to the upper folder to find a folder called logs.

       

  • akanane's avatar
    akanane
    Frequent Visitor
    2 years ago

    Hello, i am experiencing the same issue.

    Error code: 

    17:18:25,788 root ERROR Failed to initialize Spark Lighter variables. [JAVA_GATEWAY_EXITED] Java gateway process exited before sending its port number.

     

    I installed the required software and setup the JAVA_HOME and path.

    Thank you

    • Anonymous's avatar
      Anonymous
      Not applicable
      2 years ago

      Hi akanane ,
      Can you please check the syntax of JAVA_HOME environment variable?

      Please change it to %JAVA_HOME%.

      Hope this helps. Please let me know if you have any further questions. Glad to help.

      • akanane's avatar
        akanane
        Frequent Visitor
        2 years ago

        Hello Anonymous, thanks for point it out the typo.. I fixed i but still cannot use spark, this is the error code now:
        09:42:41,388 root INFO current_directory c:\Users\akanane\AppData\Local\miniconda3\772d3ce8-1d2e-4d53-b848-97351ede78a3\SynapseNotebook\4627fe3a-a5dc-4ff7-b3f5-3067b88b080f\BuildFactSales
        09:42:41,388 root INFO workspace_path c:\Users\akanane\AppData\Local\miniconda3\772d3ce8-1d2e-4d53-b848-97351ede78a3
        09:42:41,388 root INFO log_path c:\Users\akanane\AppData\Local\miniconda3\772d3ce8-1d2e-4d53-b848-97351ede78a3\logs\4627fe3a-a5dc-4ff7-b3f5-3067b88b080f
        09:42:41,388 root INFO Using synapse remote kernel ...
        09:42:41,388 root INFO Should attach session in dev mode False
        09:42:41,468 root INFO Starting session b60ba0ac-6d16-4341-9ae9-f96f55ae67ab...
        09:42:43,629 root INFO https://pbipwus13-westus.pbidedicated.windows.net/webapi/capacities/07652219-C048-4E90-B99A-868EC7DCB544/workloads/Notebook/Data/Direct/sparklighter/api/tjs/versions/2022-04-30/artifacts/4627fe3a-a5dc-4ff7-b3f5-3067b88b080f/sessions/b60ba0ac-6d16-4341-9ae9-f96f55ae67ab
        09:42:45,71 root INFO <session_management.SessionStatus object at 0x0000019A9A936800>
        09:42:45,79 root ERROR Failed to initialize Spark Lighter variables. Create session failed {'code': 'InternalError', 'subCode': 0, 'message': 'An internal error occurred.', 'timeStamp': '2023-11-02T08:42:44.9891628Z', 'httpStatusCode': 500, 'hresult': -2147467259, 'details': [{'code': 'RootActivityId', 'message': 'a038d8a0-e2e4-4375-a8ed-3dfb9866d56f'}, {'code': 'Param1', 'message': 'Response status code does not indicate success: 430 ().'}]}
        09:42:45,80 root INFO Registering Spark Lighter magics for IPython...
        09:42:45,80 root INFO Registered Spark Lighter magics for IPython.