Forum Discussion

libpekin's avatar
libpekin
Icon for Advocate II rankAdvocate II
1 year ago

Issue Ingesting Delta Sharing Tables with Deletion Vectors in Fabric Lakehouse Notebooks

1. Background:
I'm building an end‑to‑end ingestion pipeline entirely within a Microsoft Fabric Lakehouse Notebook. My source is an AWS‑hosted Delta Share, and target is OneLake staging (schema raw) followed by loads into a Fabric Warehouse.

 

2. What We’ve Tried:

  • Delta‑Sharing Spark Connector (delta-sharing-spark_2.12:0.14.0 via %%configure spark.jars.packages)

  • Delta Sharing Python client (delta-sharing PyPI package, v1.1+)

  • REST + spark.read.parquet() fallback via pre‑signed file URLs

All of these approaches work for simple Delta tables, but they fail on tables that use the Deletion Vectors feature (delta.enableDeletionVectors = true).

 

3. Error Messages:

  • REST Read → HTTP 400 with DS_UNSUPPORTED_DELTA_TABLE_FEATURES: Table features delta.enableDeletionVectors are found in table version: …

  • Delta Sharing Connector → session crashes with ClassNotFound for deltaSharing format unless a JAR is attached (which Fabric does not allow)

  • Direct Spark → unable to interpret deletion vectors, causing read failures

4. Fabric Limitations Identified While Trying to Resolve this Issue:

  1. Cannot attach external JARs or use spark.jars.packages in Lakehouse Notebooks reliably

  2. Does not support Delta Sharing REST’s "responseFormat":"delta" mode, which is required for Deletion Vectors

  3. Spark runtime in Fabric Lakehouse cannot interpret Deletion Vectors metadata in Parquet files

5. Why I Can’t Work Around on Myr Side:

  • The provider of the Delta Share is unable/unwilling to disable Deletion Vectors or rewrite tables.

  • Fabric’s current Spark runtime lacks the downstream capability to read DV‑enabled tables.

  • We have no administrative access to modify the platform’s classpath or install custom connectors.

Has anyone encountered this issue. If so, how did you resolve it? Thanks!

data engineering
notebook

8 Replies

      • v-menakakota's avatar
        v-menakakota
        Icon for Community Support rankCommunity Support
        1 year ago

        Hi libpekin  ,

        Thanks for reaching out to the Microsoft fabric community forum. can you please share the link of the post here.

        Thank you.

  • BhaveshPatel's avatar
    BhaveshPatel
    Icon for Super User rankSuper User
    1 year ago

    Then ask your fabric admins to enable Deletion Vector table properties. Use OPTIMIZE TABLE first if possible..

    • libpekin's avatar
      libpekin
      Icon for Advocate II rankAdvocate II
      1 year ago

      BhaveshPatel 

      Thank you for your response. Just to clarify—the question isn’t about configuring Deletion Vectors in Fabric. Rather, we’re seeking viable workarounds within Fabric when ingesting Delta Share tables that we don’t control, particularly when the provider has enabled Deletion Vectors and is unwilling to modify the table properties.

      What strategies exist to read or ingest from such sources given current Fabric platform limitations?

  • Tamanchu's avatar
    Tamanchu
    Icon for Super User rankSuper User
    5 months ago

    Hi,

    This is a known limitation in the current Fabric Spark runtime, it doesn't natively support reading Delta Sharing tables that have Deletion Vectors enabled. Here's why and what you can do about it.

    Why it fails:

    The Fabric Spark runtime cannot interpret Deletion Vectors metadata embedded in Parquet files. Delta Sharing's REST API also doesn't support the responseFormat: delta mode required for Deletion Vectors. And unlike Databricks, Fabric Lakehouse notebooks don't allow attaching external JARs (spark.jars.packages is restricted).

    Reference: Delta Lake table format interoperability — Microsoft Learn

    Workarounds (from most to least practical):

    Ask the data provider to create a Deletion Vector-free view/table If they can run REORG TABLE <table_name> APPLY (PURGE) on their side, it physically removes deletion vectors by rewriting Parquet files. This is the cleanest solution.

    Use an intermediate landing zone outside Fabric

    Read the Delta Sharing table from an Azure Databricks notebook (which supports Deletion Vectors natively)
    Write the result as clean Parquet/Delta to an ADLS Gen2 location
    Use a Fabric Shortcut or Copy Activity to bring it into your Lakehouse
    # In Databricks notebook
    df = spark.read.format("deltaSharing").load("<profile>#<share>.<schema>.<table>")
    df.write.format("delta").mode("overwrite").save("abfss://<container>@<storage>.dfs.core.windows.net/clean_landing/")
    Then in Fabric, create a OneLake shortcut pointing to that ADLS Gen2 path.

    Use Fabric Data Pipeline with Copy Activity Instead of notebooks, use a Pipeline Copy Activity with the source configured as the external storage where the provider can export data. This bypasses the Spark runtime entirely.

    Submit a feature request on Fabric Ideas (which you already did — great move!) The Fabric team is actively working on expanding Delta Lake feature support. Deletion Vectors support in the Spark runtime is one of the most requested features.

    Reference: Delta Table Maintenance in Microsoft Fabric

    Hope this helps! Give a kudos. Let me know if you need more details on the intermediate landing zone approach.