Issue Ingesting Delta Sharing Tables with Deletion Vectors in Fabric Lakehouse Notebooks

[libpekin]

1. Background:
I'm building an end‑to‑end ingestion pipeline entirely within a Microsoft Fabric Lakehouse Notebook. My source is an AWS‑hosted Delta Share, and target is OneLake staging (schema raw) followed by loads into a Fabric Warehouse.

2. What We’ve Tried:

• Delta‑Sharing Spark Connector (delta-sharing-spark_2.12:0.14.0 via %%configure spark.jars.packages)

• Delta Sharing Python client (delta-sharing PyPI package, v1.1+)

• REST + spark.read.parquet() fallback via pre‑signed file URLs

All of these approaches work for simple Delta tables, but they fail on tables that use the Deletion Vectors feature (delta.enableDeletionVectors = true).

3. Error Messages:

• REST Read → HTTP 400 with DS_UNSUPPORTED_DELTA_TABLE_FEATURES: Table features delta.enableDeletionVectors are found in table version: …

• Delta Sharing Connector → session crashes with ClassNotFound for deltaSharing format unless a JAR is attached (which Fabric does not allow)

• Direct Spark → unable to interpret deletion vectors, causing read failures

4. Fabric Limitations Identified While Trying to Resolve this Issue:

1. Cannot attach external JARs or use spark.jars.packages in Lakehouse Notebooks reliably

2. Does not support Delta Sharing REST’s "responseFormat":"delta" mode, which is required for Deletion Vectors

3. Spark runtime in Fabric Lakehouse cannot interpret Deletion Vectors metadata in Parquet files

5. Why I Can’t Work Around on Myr Side:

• The provider of the Delta Share is unable/unwilling to disable Deletion Vectors or rewrite tables.

• Fabric’s current Spark runtime lacks the downstream capability to read DV‑enabled tables.

• We have no administrative access to modify the platform’s classpath or install custom connectors.

Has anyone encountered this issue. If so, how did you resolve it? Thanks!

[BhaveshPatel]

Then ask your fabric admins to enable Deletion Vector table properties. Use OPTIMIZE TABLE first if possible..

[lbendlin]

If this is important to you please consider voting for an existing idea or raising a new one at https://ideas.fabric.microsoft.com

[libpekin]

This post has been added to Fabric Ideas. @lbendlin @BhaveshPatel please consider voting for this idea. Thanks!

[v-menakakota]

Hi  @libpekin ,

Thank you for sharing your idea in the Ideas forum.

We appreciate your contribution. Please note that Microsoft regularly reviews ideas shared by the community, and your suggestion may be considered for future updates.

Thank you once again, and we encourage you to continue engaging with the Microsoft Fabric Community.

Best regards,
Community Support Team.

[v-menakakota]

Hi @libpekin  ,

Thanks for reaching out to the Microsoft fabric community forum. can you please share the link of the post here.

Thank you.

[BhaveshPatel]

Try single delta table at a time. It is working in my scenario. Deletion Vectors..It is more advanced version of OPTIMIZE ( Delta Lake Deletion Vectors | Delta Lake ). Also, Try creating a delta table first...it becomes easy then..

[libpekin]

@BhaveshPatel 

Thank you for your response. Just to clarify—the question isn’t about configuring Deletion Vectors in Fabric. Rather, we’re seeking viable workarounds within Fabric when ingesting Delta Share tables that we don’t control, particularly when the provider has enabled Deletion Vectors and is unwilling to modify the table properties.

What strategies exist to read or ingest from such sources given current Fabric platform limitations?