topic Re: VSCode - Connecting to lakehouse and coding aid in Data Engineering

VSCode - Connecting to lakehouse and coding aid

kjellvs — Sat, 21 Oct 2023 09:02:52 GMT

When running simple code in VSCode using the Synapse-spark-kernel, i run into the following issue when trying to access data from a lakehouse table. The exact same code does work in the fabric environment.

spark_df = spark.sql("select * from LakeHouse.table")

java.lang.ClassNotFoundException: Failed to find data source: LakeHouse. Please find packages at https://spark.apache.org/third-party-projects.html

How do I connect to the lakehouse in local VSCode via the synapse-spark-kernel?

Furthermore, I was wondering if there are any recommended VSCode extensions for code validation and aid (like intellisense), that work with the synapse-spark-kernel?

kind regards,

Kjell

Re: VSCode - Connecting to lakehouse and coding aid

Anonymous — Sat, 21 Oct 2023 11:19:49 GMT

Hi @kjellvs ,
Thanks for using Fabric Community.

Apologies for the issue you have been facing. We need to add lakehouse into the notebook in Fabric portal, then you can use this lakehouse in Synapse VScode extension without any error.
You can follow the below steps:

1) Add the lakehouse to the respective notebook.

2) After adding the lakehouse in Fabric portal, please download the Notebook again, because the notebook's metadata has changed, and VS Code extension can't get the change. So download the notebook once again.

3) Try running the code now in your notebook.

If you are using python, you can install python extension which provides excellent language services and auto complete capability.

Hope this helps. Please let us know if you have any further queries. Glad to help.

Re: VSCode - Connecting to lakehouse and coding aid

Anonymous — Mon, 23 Oct 2023 09:52:29 GMT

Hi @kjellvs ,
We haven’t heard from you on the last response and was just checking back to see if your query has been resolved. Please let us know if you have any further queries.

Re: VSCode - Connecting to lakehouse and coding aid

kjellvs — Tue, 24 Oct 2023 11:27:50 GMT

hi @Anonymous,

Thank you for your swift reply. I took a bit longer over the weekend.

The first problem seems to be solved for the "defaultLakehouse", i.e., I am able to query tables from the lakehouse configured in fabric, shown next to the two arrows. However, I have two lakehouses currently referenced in my notebook (Raw data and processed data). I would like to access both from the notebook, however this does not seem possible, as I still get the similar error for the referenced lakehouse, which is not the default (java.lang.ClassNotFoundException: Failed to find data source: RawData. Please find packages at).

(metadata of the notebook according to vscode).

Is there a way to access both?

Besides this, I was also running into an issue where the kernel wasn't properly initialized. I am checking this by printing the "spark" variable. Restarting kernal/vscode sometimes was necessary for complete fix. If there are any other ways to fix this, let me know!

For the extension, you are right, I am using this one. I added the following at the top to make the extension also understand the predefined variables.

from pyspark.sql import SparkSession spark: SparkSession

Hope you can help me further.

Kjell

Re: VSCode - Connecting to lakehouse and coding aid

Anonymous — Wed, 25 Oct 2023 06:59:23 GMT

Hi @kjellvs ,
Apologies for the delay in response. This error is expected behavior with the existing version of Fabric lakehouse. I filed a bug for reading reference lakehouse in VS Code and the team is working on fixing the issue ASAP.
We are reaching out to the internal team to get more information related to ETA and will get back to you as soon as we have an update.
Stay tuned for more updates.
Thanks.

Re: VSCode - Connecting to lakehouse and coding aid

Anonymous — Thu, 26 Oct 2023 15:04:17 GMT

Hi @kjellvs ,
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. Please let me know if you have any further queries.

Re: VSCode - Connecting to lakehouse and coding aid

kjellvs — Thu, 26 Oct 2023 15:06:57 GMT

hi @Anonymous,

as I understood from your previous response, there is no resolution yet as your team is working on fixing the issue ASAP, and that I should stay tuned. I guess this answers my questions, should I close the ticket in the mean time? Where could I find an update on this once this has progressed?

Kind regards,
Kjell

Re: VSCode - Connecting to lakehouse and coding aid

Anonymous — Thu, 26 Oct 2023 15:09:16 GMT

Hi @kjellvs ,
I will update you once I receive the information from the internal team. You can go ahead and close the ticket.
Thank you.

Re: VSCode - Connecting to lakehouse and coding aid

y0m0 — Thu, 23 Nov 2023 12:18:58 GMT

I am not the OP but the issue is not solved yet.
Your suggested solution works if there is only 1 lakehouse reference in the notebook and if its marked as default.
Like OP described, in a notebook I am trying to query data from 2 different lakehouse within the same workspace.

For example I have this 2 lakehouse registered on my notebook in fabric.

This works as expected from the web editor.

the same is not working from VSCode.

Re: VSCode - Connecting to lakehouse and coding aid

kjellvs — Thu, 23 Nov 2023 12:23:23 GMT

as mentioned by support;

"I will update you once I receive the information from the internal team. You can go ahead and close the ticket."