Forum Discussion

lizle's avatar
lizle
Regular Visitor
3 months ago
Solved

Reading data from Files folder in VS Code Fabric Data Engineering Extension Locally

Hi, I'm trying to read data from textfile into a dataframe using spark in the VS Code extension of Fabric locally. I have two lakehouses in my notebook. This is my code:  try:     df = spark.read....
  • Parchitect's avatar
    3 months ago

    Hi,

    I have tried to recreate the problem you are facing.
    The quickest way to fix this right now is to bypass local paths entirely and use the **ABFS path** directly from the UI:

    1.  Go to the Fabric Data Engineering explorer pane on the left side in VS Code.
    2. Right-click your `datpost.txt` file and select **Copy ABFS Path**.
      Example from picture from my VSCode:

       

    3.  Paste that full string directly into your Notebook:
    df = spark.read.text("**Copy ABFS Path**")
    df.show(10)​


    Note that when you copy the abfs path, you get the start of the string "abfss" instead of "abfs", reason is:
    Abfss (abfss://) forces the use of TLS (Transport Layer Security) for secure, encrypted data transfer over the network, whereas abfs (abfs://) is not enforching TLS.

    Now to why you get this problem:
    When you use the VS Code Fabric extension against a remote Spark compute(Fabric), the code actually executes inside a remote container in Azure. That container has no awareness of your local Windows or VS Code folder structure (/Concept/Notebooks/...), causing Spark to get stuck in an infinite timeout loop trying to find a directory that doesn't exist on the cluster.

    A better way for production notebook without hardcoded ADFS:

    If you want to avoid hardcoding long ABFS strings in your code, you can use mssparkutils to resolve the cloud path dynamically. 

     

    from notebookutils import mssparkutils
    
    # Fetch the path dynamically (replace "bronze_lakehouse" with your actual lakehouse name)
    lakehouse_path = mssparkutils.lakehouse.get("bronze_lakehouse").properties["abfsPath"]
    
    # Read the file using the dynamic cloud path
    df = spark.read.text(f"{lakehouse_path}/Files/Billing/Post/datpost.txt")
    df.show(10)

     

    Best regards,

    Parchitect - Solutions Architect

    💡Did my response help you? Clicking Kudos is a small gesture that goes a long way!

    ✔️Did I answer your question? Please mark my post as a Solution to help others find it faster.