Forum Discussion

kjellvs's avatar
kjellvs
Helper I
2 years ago
Solved

VSCode - Connecting to lakehouse and coding aid

When running simple code in VSCode using the Synapse-spark-kernel, i run into the following issue when trying to access data from a lakehouse table. The exact same code does work in the fabric environment.

 

spark_df = spark.sql("select * from LakeHouse.table")

 

java.lang.ClassNotFoundException: Failed to find data source: LakeHouse. Please find packages at https://spark.apache.org/third-party-projects.html

 

How do I connect to the lakehouse in local VSCode via the synapse-spark-kernel?

Furthermore, I was wondering if there are any recommended VSCode extensions for code validation and aid (like intellisense), that work with the synapse-spark-kernel?

 

kind regards,

Kjell

data engineering
  • Anonymous's avatar
    Anonymous
    2 years ago

    Hi kjellvs ,
    Apologies for the delay in response. This error is expected behavior with the existing version of Fabric lakehouse. I filed a bug for reading reference lakehouse in VS Code and the team is working on fixing the issue ASAP.
    We are reaching out to the internal team to get more information related to ETA and will get back to you as soon as we have an update.
    Stay tuned for more updates.
    Thanks.



  • Anonymous's avatar
    Anonymous
    2 years ago

    Hi kjellvs ,
    I will update you once I receive the information from the internal team. You can go ahead and close the ticket.
    Thank you.

9 Replies

  • Anonymous's avatar
    Anonymous
    Not applicable
    2 years ago

    Hi kjellvs ,
    Thanks for using Fabric Community.

    Apologies for the issue you have been facing. We need to add lakehouse into the notebook in Fabric portal, then you can use this lakehouse in Synapse VScode extension without any error.
    You can follow the below steps:

    1) Add the lakehouse to the respective notebook.

     


    2) After adding the lakehouse in Fabric portal, please download the Notebook again, because the notebook's metadata has changed, and VS Code extension can't get the change. So download the notebook once again.

    3) Try running the code now in your notebook. 


    If you are using python, you can install python extension which provides excellent language services and auto complete capability.


    Hope this helps. Please let us know if you have any further queries. Glad to help.

    • Anonymous's avatar
      Anonymous
      Not applicable
      2 years ago

      Hi kjellvs ,
      We haven’t heard from you on the last response and was just checking back to see if your query has been resolved. Please let us know if you have any further queries.

  • kjellvs's avatar
    kjellvs
    Helper I
    2 years ago

    hi Anonymous, 

    Thank you for your swift reply. I took a bit longer over the weekend.

    The first problem seems to be solved for the "defaultLakehouse", i.e., I am able to query tables from the lakehouse configured in fabric, shown next to the two arrows. However, I have two lakehouses currently referenced in my notebook (Raw data and processed data). I would like to access both from the notebook, however this does not seem possible, as I still get the similar error for the referenced lakehouse, which is not the default (java.lang.ClassNotFoundException: Failed to find data source: RawData. Please find packages at).

    (metadata of the notebook according to vscode).

    Is there a way to access both?

     

    Besides this, I was also running into an issue where the kernel wasn't properly initialized. I am checking this by printing the "spark" variable. Restarting kernal/vscode sometimes was necessary for complete fix. If there are any other ways to fix this, let me know!

     

    For the extension, you are right, I am using this one. I added the following at the top to make the extension also understand the predefined variables. 

     

    from pyspark.sql import SparkSession
spark: SparkSession

    Hope you can help me further.

    Kjell

     

    • Anonymous's avatar
      Anonymous
      Not applicable
      2 years ago

      Hi kjellvs ,
      Apologies for the delay in response. This error is expected behavior with the existing version of Fabric lakehouse. I filed a bug for reading reference lakehouse in VS Code and the team is working on fixing the issue ASAP.
      We are reaching out to the internal team to get more information related to ETA and will get back to you as soon as we have an update.
      Stay tuned for more updates.
      Thanks.



      • Anonymous's avatar
        Anonymous
        Not applicable
        2 years ago

        Hi kjellvs ,
        We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. Please let me know if you have any further queries.

  • y0m0's avatar
    y0m0
    Frequent Visitor
    2 years ago

    I am not the OP but the issue is not solved yet.
    Your suggested solution works if there is only 1 lakehouse reference in the notebook and if its marked as default.
    Like OP described, in a notebook I am trying to query data from 2 different lakehouse within the same workspace.

    For example I have this 2 lakehouse registered on my notebook in fabric.

    This works as expected from the web editor.


    the same is not working from VSCode.

    • kjellvs's avatar
      kjellvs
      Helper I
      2 years ago

      as mentioned by support; 

      "I will update you once I receive the information from the internal team. You can go ahead and close the ticket."