Forum Discussion
Reading data from Files folder in VS Code Fabric Data Engineering Extension Locally
- 3 months ago
Hi,
I have tried to recreate the problem you are facing.
The quickest way to fix this right now is to bypass local paths entirely and use the **ABFS path** directly from the UI:- Go to the Fabric Data Engineering explorer pane on the left side in VS Code.
- Right-click your `datpost.txt` file and select **Copy ABFS Path**.
Example from picture from my VSCode: - Paste that full string directly into your Notebook:
df = spark.read.text("**Copy ABFS Path**") df.show(10)
Note that when you copy the abfs path, you get the start of the string "abfss" instead of "abfs", reason is:
Abfss (abfss://) forces the use of TLS (Transport Layer Security) for secure, encrypted data transfer over the network, whereas abfs (abfs://) is not enforching TLS.Now to why you get this problem:
When you use the VS Code Fabric extension against a remote Spark compute(Fabric), the code actually executes inside a remote container in Azure. That container has no awareness of your local Windows or VS Code folder structure (/Concept/Notebooks/...), causing Spark to get stuck in an infinite timeout loop trying to find a directory that doesn't exist on the cluster.
A better way for production notebook without hardcoded ADFS:If you want to avoid hardcoding long ABFS strings in your code, you can use mssparkutils to resolve the cloud path dynamically.
from notebookutils import mssparkutils # Fetch the path dynamically (replace "bronze_lakehouse" with your actual lakehouse name) lakehouse_path = mssparkutils.lakehouse.get("bronze_lakehouse").properties["abfsPath"] # Read the file using the dynamic cloud path df = spark.read.text(f"{lakehouse_path}/Files/Billing/Post/datpost.txt") df.show(10)Best regards,
Parchitect - Solutions Architect
💡Did my response help you? Clicking Kudos is a small gesture that goes a long way!
✔️Did I answer your question? Please mark my post as a Solution to help others find it faster.
Hi,
I have tried to recreate the problem you are facing.
The quickest way to fix this right now is to bypass local paths entirely and use the **ABFS path** directly from the UI:
- Go to the Fabric Data Engineering explorer pane on the left side in VS Code.
- Right-click your `datpost.txt` file and select **Copy ABFS Path**.
Example from picture from my VSCode: - Paste that full string directly into your Notebook:
df = spark.read.text("**Copy ABFS Path**")
df.show(10)
Note that when you copy the abfs path, you get the start of the string "abfss" instead of "abfs", reason is:
Abfss (abfss://) forces the use of TLS (Transport Layer Security) for secure, encrypted data transfer over the network, whereas abfs (abfs://) is not enforching TLS.
Now to why you get this problem:
When you use the VS Code Fabric extension against a remote Spark compute(Fabric), the code actually executes inside a remote container in Azure. That container has no awareness of your local Windows or VS Code folder structure (/Concept/Notebooks/...), causing Spark to get stuck in an infinite timeout loop trying to find a directory that doesn't exist on the cluster.
A better way for production notebook without hardcoded ADFS:
If you want to avoid hardcoding long ABFS strings in your code, you can use mssparkutils to resolve the cloud path dynamically.
from notebookutils import mssparkutils
# Fetch the path dynamically (replace "bronze_lakehouse" with your actual lakehouse name)
lakehouse_path = mssparkutils.lakehouse.get("bronze_lakehouse").properties["abfsPath"]
# Read the file using the dynamic cloud path
df = spark.read.text(f"{lakehouse_path}/Files/Billing/Post/datpost.txt")
df.show(10)
Best regards,
Parchitect - Solutions Architect
💡Did my response help you? Clicking Kudos is a small gesture that goes a long way!
✔️Did I answer your question? Please mark my post as a Solution to help others find it faster.