Forum Discussion

dave_fackler's avatar
dave_fackler
New Member
11 months ago
Solved

Error Reading Data from Mirrored Database Using Lakehouse Shortcut

I have a mirrored database set up in a trial Fabric workspace. The data is coming from an Azure SQL database. Works great. I have a lakehouse in the same workspace and I created a shortcut to the mirrored database in the lakehouse. So all of the mirrored tables now show up under the Files list in the lakehouse. I'm trying to do a very simple read of the data using a notebook and this code:

 

# Read Delta table using Spark
spark_df = spark.read.format("delta").load("/[lakehousename]/default/Files/[shortcutname]/[tablename]")
 
When I run this, I get a Spark_System_ABFS_OperationFailed error with this error message:
 
An operation with ADLS Gen2 has failed. This is typically due to a permissions issue. 1. Please ensure that for all ADLS Gen2 resources referenced in the Spark job, that the user running the code has RBAC roles "Storage Blob Data Contributor" on storage accounts the job is expected to read and write from. 2. Check the logs for this Spark application. Inspect the logs for the ADLS Gen2 storage account name that is experiencing this issue.
 
I am an admin in the workspace and have all rights to the mirrored database and the lakehouse (from what I can tell). Is there a permission somewhere that I'm missing? Or is this just not a supported method for reading the mirrored database data from a notebook?
  • Hi dave_fackler

    This is supported, but you need the correct path and the right Fabric/OneLake permissions.

     

    The error text mentions “Storage Blob Data Contributor,” but with Fabric OneLake, access is governed by Fabric item/OneLake permissions rather than Azure Storage RBAC. For mirrored databases, make sure your account has Read all OneLake data on the mirrored database (Workspace ->the mirrored database -> Share). That permission explicitly allows accessing the mirrored data files from Spark/OneLake Explorer. Docs: Share your mirrored database & permissions

    It’s supported to read mirrored data from notebooks

    Mirroring lands your source tables as Delta files in OneLake. You can explore/query them with Spark in notebooks by creating a Lakehouse shortcut to the mirrored database/tables and reading from it. Docs: Explore mirrored data with notebooks - Mirroring overview

    Use the correct path (copy the ABFS path from the UI)

    1. In your Lakehouse, right-click the shortcut (or a table/file in it) -> Properties -> copy the ABFS path (or “Copy relative path for Spark” if it’s your default Lakehouse). Docs: Copy ABFS/relative paths
    2. Use that ABFS path in your notebook.
    # Example: read a mirrored Delta table through a Lakehouse shortcut (ABFS path)
    path = "abfss://<workspaceName>@onelake.dfs.fabric.microsoft.com/<lakehouseName>.lakehouse/Files/<shortcutName>/<schema>/<tableName>"
    df = spark.read.format("delta").load(path)
    display(df)

    ABFS path format example (from Microsoft docs): abfss://[email protected]/myLakehouse.lakehouse/Files/. Docs: OneLake ABFSS path example

    Tip: ensure the path targets the actual table folder that contains a _delta_log (that’s the Delta table root). Higher-level folders (schema/database only) won’t load as Delta tables. Delta read-by-path points to folder with _delta_log Delta logs live under _delta_log

    Even easier: add the shortcut under Tables

    If the target is Delta, create the shortcut in the Lakehouse Tables section (not Files). Fabric auto-discovers it as a table so you can read it like any managed table via Spark/SQL/semantic model.

    # Read a table-shortcut registered under Tables
    df = spark.read.table("dbo.MyShortcutTable")            # Spark
    # or
    spark.sql("SELECT * FROM dbo.MyShortcutTable").show()   # SQL

    Docs: OneLake shortcuts (Tables vs Files, auto-recognition) - Lakehouse & Delta tables (auto-discovery incl. shortcuts)

    Quick checklist

    • Grant yourself Read all OneLake data on the mirrored database (Share -> permissions). Docs
    • Copy the shortcut’s ABFS path (or relative path) and use it in spark.read.format("delta").load(...). Docs
    • Point to the actual table folder (must contain _delta_log). Docs
    • For Delta tables, prefer the shortcut in Tables so it’s addressable as a table. Docs

    If you found this helpful, consider giving some Kudos. If I answered your question or solved your problem, please mark this as the solution.

4 Replies

  • Hi dave_fackler

    This is supported, but you need the correct path and the right Fabric/OneLake permissions.

     

    The error text mentions “Storage Blob Data Contributor,” but with Fabric OneLake, access is governed by Fabric item/OneLake permissions rather than Azure Storage RBAC. For mirrored databases, make sure your account has Read all OneLake data on the mirrored database (Workspace ->the mirrored database -> Share). That permission explicitly allows accessing the mirrored data files from Spark/OneLake Explorer. Docs: Share your mirrored database & permissions

    It’s supported to read mirrored data from notebooks

    Mirroring lands your source tables as Delta files in OneLake. You can explore/query them with Spark in notebooks by creating a Lakehouse shortcut to the mirrored database/tables and reading from it. Docs: Explore mirrored data with notebooks - Mirroring overview

    Use the correct path (copy the ABFS path from the UI)

    1. In your Lakehouse, right-click the shortcut (or a table/file in it) -> Properties -> copy the ABFS path (or “Copy relative path for Spark” if it’s your default Lakehouse). Docs: Copy ABFS/relative paths
    2. Use that ABFS path in your notebook.
    # Example: read a mirrored Delta table through a Lakehouse shortcut (ABFS path)
    path = "abfss://<workspaceName>@onelake.dfs.fabric.microsoft.com/<lakehouseName>.lakehouse/Files/<shortcutName>/<schema>/<tableName>"
    df = spark.read.format("delta").load(path)
    display(df)

    ABFS path format example (from Microsoft docs): abfss://[email protected]/myLakehouse.lakehouse/Files/. Docs: OneLake ABFSS path example

    Tip: ensure the path targets the actual table folder that contains a _delta_log (that’s the Delta table root). Higher-level folders (schema/database only) won’t load as Delta tables. Delta read-by-path points to folder with _delta_log Delta logs live under _delta_log

    Even easier: add the shortcut under Tables

    If the target is Delta, create the shortcut in the Lakehouse Tables section (not Files). Fabric auto-discovers it as a table so you can read it like any managed table via Spark/SQL/semantic model.

    # Read a table-shortcut registered under Tables
    df = spark.read.table("dbo.MyShortcutTable")            # Spark
    # or
    spark.sql("SELECT * FROM dbo.MyShortcutTable").show()   # SQL

    Docs: OneLake shortcuts (Tables vs Files, auto-recognition) - Lakehouse & Delta tables (auto-discovery incl. shortcuts)

    Quick checklist

    • Grant yourself Read all OneLake data on the mirrored database (Share -> permissions). Docs
    • Copy the shortcut’s ABFS path (or relative path) and use it in spark.read.format("delta").load(...). Docs
    • Point to the actual table folder (must contain _delta_log). Docs
    • For Delta tables, prefer the shortcut in Tables so it’s addressable as a table. Docs

    If you found this helpful, consider giving some Kudos. If I answered your question or solved your problem, please mark this as the solution.

  • v-venuppu's avatar
    v-venuppu
    Community Support

    Hi dave_fackler ,

    Thank you for reaching out to Microsoft Fabric Community.

    Thank you tayloramy for the prompt response.

    I wanted to check if you had the opportunity to review the information provided and resolve the issue..?Please let us know if you need any further assistance.We are happy to help.

    Thank you.

    • tayloramy's avatar
      tayloramy
      Super User

      Hi dave_fackler

      Glad I could help. If the problem is resolved, kindly mark the post as the solution so that other forum users can easily find it in the future. 

       

      If you found this helpful, consider giving some Kudos. If I answered your question or solved your problem, please mark this as the solution.