Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

Register now to learn Fabric in free live sessions led by the best Microsoft experts. From Apr 16 to May 9, in English and Spanish.

Reply
kjellvs
Helper I
Helper I

VSCode - Connecting to lakehouse and coding aid

When running simple code in VSCode using the Synapse-spark-kernel, i run into the following issue when trying to access data from a lakehouse table. The exact same code does work in the fabric environment.

 

spark_df = spark.sql("select * from LakeHouse.table")

 

java.lang.ClassNotFoundException: Failed to find data source: LakeHouse. Please find packages at https://spark.apache.org/third-party-projects.html

 

How do I connect to the lakehouse in local VSCode via the synapse-spark-kernel?

Furthermore, I was wondering if there are any recommended VSCode extensions for code validation and aid (like intellisense), that work with the synapse-spark-kernel?

 

kind regards,

Kjell

2 ACCEPTED SOLUTIONS

Hi @kjellvs ,
Apologies for the delay in response. This error is expected behavior with the existing version of Fabric lakehouse. I filed a bug for reading reference lakehouse in VS Code and the team is working on fixing the issue ASAP.
We are reaching out to the internal team to get more information related to ETA and will get back to you as soon as we have an update.
Stay tuned for more updates.
Thanks.



View solution in original post

Hi @kjellvs ,
I will update you once I receive the information from the internal team. You can go ahead and close the ticket.
Thank you.

View solution in original post

9 REPLIES 9
y0m0
Frequent Visitor

I am not the OP but the issue is not solved yet.
Your suggested solution works if there is only 1 lakehouse reference in the notebook and if its marked as default.
Like OP described, in a notebook I am trying to query data from 2 different lakehouse within the same workspace.

For example I have this 2 lakehouse registered on my notebook in fabric.

y0m0_0-1700739390940.png

This works as expected from the web editor.

y0m0_3-1700740547630.png


the same is not working from VSCode.

as mentioned by support; 

"I will update you once I receive the information from the internal team. You can go ahead and close the ticket."

kjellvs
Helper I
Helper I

hi @v-nikhilan-msft

Thank you for your swift reply. I took a bit longer over the weekend.

The first problem seems to be solved for the "defaultLakehouse", i.e., I am able to query tables from the lakehouse configured in fabric, shown next to the two arrows. However, I have two lakehouses currently referenced in my notebook (Raw data and processed data). I would like to access both from the notebook, however this does not seem possible, as I still get the similar error for the referenced lakehouse, which is not the default (java.lang.ClassNotFoundException: Failed to find data source: RawData. Please find packages at).

kjellvs_1-1698146547652.png

(metadata of the notebook according to vscode).

Is there a way to access both?

 

Besides this, I was also running into an issue where the kernel wasn't properly initialized. I am checking this by printing the "spark" variable. Restarting kernal/vscode sometimes was necessary for complete fix. If there are any other ways to fix this, let me know!

 

For the extension, you are right, I am using this one. I added the following at the top to make the extension also understand the predefined variables. 

 

from pyspark.sql import SparkSession
spark: SparkSession 

Hope you can help me further.

Kjell

 

Hi @kjellvs ,
Apologies for the delay in response. This error is expected behavior with the existing version of Fabric lakehouse. I filed a bug for reading reference lakehouse in VS Code and the team is working on fixing the issue ASAP.
We are reaching out to the internal team to get more information related to ETA and will get back to you as soon as we have an update.
Stay tuned for more updates.
Thanks.



Hi @kjellvs ,
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. Please let me know if you have any further queries.

hi @v-nikhilan-msft,

as I understood from your previous response, there is no resolution yet as your team is working on fixing the issue ASAP, and that I should stay tuned. I guess this answers my questions, should I close the ticket in the mean time? Where could I find an update on this once this has progressed?

Kind regards,
Kjell

Hi @kjellvs ,
I will update you once I receive the information from the internal team. You can go ahead and close the ticket.
Thank you.

v-nikhilan-msft
Community Support
Community Support

Hi @kjellvs ,
Thanks for using Fabric Community.

Apologies for the issue you have been facing. We need to add lakehouse into the notebook in Fabric portal, then you can use this lakehouse in Synapse VScode extension without any error.
You can follow the below steps:

1) Add the lakehouse to the respective notebook.

vnikhilanmsft_0-1697886580850.png

vnikhilanmsft_3-1697886964424.png

 


2) After adding the lakehouse in Fabric portal, please download the Notebook again, because the notebook's metadata has changed, and VS Code extension can't get the change. So download the notebook once again.

3) Try running the code now in your notebook. 

vnikhilanmsft_2-1697886812052.png


If you are using python, you can install python extension which provides excellent language services and auto complete capability.

vnikhilanmsft_4-1697887040441.png


Hope this helps. Please let us know if you have any further queries. Glad to help.

Hi @kjellvs ,
We haven’t heard from you on the last response and was just checking back to see if your query has been resolved. Please let us know if you have any further queries.

Helpful resources

Announcements
Microsoft Fabric Learn Together

Microsoft Fabric Learn Together

Covering the world! 9:00-10:30 AM Sydney, 4:00-5:30 PM CET (Paris/Berlin), 7:00-8:30 PM Mexico City

March 2024  FBC Gallery Image

Fabric Monthly Update - March 2024

Check out the March 2024 Fabric update to learn about new features.

April Fabric Community Update

Fabric Community Update - April 2024

Find out what's new and trending in the Fabric Community.

Top Kudoed Authors