Forum Discussion
Reading data from Files folder in VS Code Fabric Data Engineering Extension Locally
Hi, I'm trying to read data from textfile into a dataframe using spark in the VS Code extension of Fabric locally. I have two lakehouses in my notebook. This is my code:
Hi,
I have tried to recreate the problem you are facing.
The quickest way to fix this right now is to bypass local paths entirely and use the **ABFS path** directly from the UI:- Go to the Fabric Data Engineering explorer pane on the left side in VS Code.
- Right-click your `datpost.txt` file and select **Copy ABFS Path**.
Example from picture from my VSCode: - Paste that full string directly into your Notebook:
df = spark.read.text("**Copy ABFS Path**") df.show(10)
Note that when you copy the abfs path, you get the start of the string "abfss" instead of "abfs", reason is:
Abfss (abfss://) forces the use of TLS (Transport Layer Security) for secure, encrypted data transfer over the network, whereas abfs (abfs://) is not enforching TLS.Now to why you get this problem:
When you use the VS Code Fabric extension against a remote Spark compute(Fabric), the code actually executes inside a remote container in Azure. That container has no awareness of your local Windows or VS Code folder structure (/Concept/Notebooks/...), causing Spark to get stuck in an infinite timeout loop trying to find a directory that doesn't exist on the cluster.
A better way for production notebook without hardcoded ADFS:If you want to avoid hardcoding long ABFS strings in your code, you can use mssparkutils to resolve the cloud path dynamically.
from notebookutils import mssparkutils # Fetch the path dynamically (replace "bronze_lakehouse" with your actual lakehouse name) lakehouse_path = mssparkutils.lakehouse.get("bronze_lakehouse").properties["abfsPath"] # Read the file using the dynamic cloud path df = spark.read.text(f"{lakehouse_path}/Files/Billing/Post/datpost.txt") df.show(10)Best regards,
Parchitect - Solutions Architect
💡Did my response help you? Clicking Kudos is a small gesture that goes a long way!
✔️Did I answer your question? Please mark my post as a Solution to help others find it faster.
2 Replies
- V-yubandi-msftCommunity Support
Hi lizle ,
If you get a chance, please review the response shared by Parchitect and let us know if it aligns with your expectations. Should you need any additional details or clarification, feel free to let us know.Thank you for your valuable input Parchitect .
Regards,
Yugandhar. - ParchitectSolution Sage
Hi,
I have tried to recreate the problem you are facing.
The quickest way to fix this right now is to bypass local paths entirely and use the **ABFS path** directly from the UI:- Go to the Fabric Data Engineering explorer pane on the left side in VS Code.
- Right-click your `datpost.txt` file and select **Copy ABFS Path**.
Example from picture from my VSCode: - Paste that full string directly into your Notebook:
df = spark.read.text("**Copy ABFS Path**") df.show(10)
Note that when you copy the abfs path, you get the start of the string "abfss" instead of "abfs", reason is:
Abfss (abfss://) forces the use of TLS (Transport Layer Security) for secure, encrypted data transfer over the network, whereas abfs (abfs://) is not enforching TLS.Now to why you get this problem:
When you use the VS Code Fabric extension against a remote Spark compute(Fabric), the code actually executes inside a remote container in Azure. That container has no awareness of your local Windows or VS Code folder structure (/Concept/Notebooks/...), causing Spark to get stuck in an infinite timeout loop trying to find a directory that doesn't exist on the cluster.
A better way for production notebook without hardcoded ADFS:If you want to avoid hardcoding long ABFS strings in your code, you can use mssparkutils to resolve the cloud path dynamically.
from notebookutils import mssparkutils # Fetch the path dynamically (replace "bronze_lakehouse" with your actual lakehouse name) lakehouse_path = mssparkutils.lakehouse.get("bronze_lakehouse").properties["abfsPath"] # Read the file using the dynamic cloud path df = spark.read.text(f"{lakehouse_path}/Files/Billing/Post/datpost.txt") df.show(10)Best regards,
Parchitect - Solutions Architect
💡Did my response help you? Clicking Kudos is a small gesture that goes a long way!
✔️Did I answer your question? Please mark my post as a Solution to help others find it faster.