Forum Discussion
VS Code Integration. Spark.SQL Query working whereas Relative Filepath is not
Hi matthewstainsby
We don't support mssparkutils or mount lakehouse in VS Code currently, so you cannot do these file system operations in our extension. You can do these concat operations on Fabric Portal, and then read these data in VSCode.
Here is Lakehouse public api:
Items - List Lakehouses - REST API (Lakehouse) | Microsoft Learn
You can use this to list lakehouses files too.
Hope this helps. Please let me know if you have any further questions.
Hello and thank you for the response
The problem is that I have a lot of files (>500), so using the approach you suggested would not be possible.
Reading the list would allow me to do something like this, which would append all the dfs together.
If I can read out the list of files it could make our ETL in VS code possible.
dfs = []
for f_path in f_paths:
dfs.append(spark.read.format("csv").option("header","true").load(f_path))
df_combined = df.concat(dfs)
Hi matthewstainsby
We don't support mssparkutils or mount lakehouse in VS Code currently, so you cannot do these file system operations in our extension. You can do these concat operations on Fabric Portal, and then read these data in VSCode.
Here is Lakehouse public api:
Items - List Lakehouses - REST API (Lakehouse) | Microsoft Learn
You can use this to list lakehouses files too.
Hope this helps. Please let me know if you have any further questions.
Thank you so much, I'll look into using the API and open another thread if I can't find a solution.
Hi matthewstainsby
Glad that your query got resolved. Please continue using Fabric Community for any help regarding your queries.Hi matthewstainsby
Glad that your query got resolved. Please continue using Fabric Community for any help regarding your queries.
Hi @matthewstainsby
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. Otherwise, will respond back with the more details and we will try to help.
Thanks