topic Re: Issue Ingesting Delta Sharing Tables with Deletion Vectors in Fabric Lakehouse Notebooks in Data Engineering

Issue Ingesting Delta Sharing Tables with Deletion Vectors in Fabric Lakehouse Notebooks

libpekin — Fri, 18 Jul 2025 21:48:30 GMT

1. Background:
I'm building an end‑to‑end ingestion pipeline entirely within a Microsoft Fabric Lakehouse Notebook. My source is an AWS‑hosted Delta Share, and target is OneLake staging (schema raw) followed by loads into a Fabric Warehouse.

2. What We’ve Tried:

Delta‑Sharing Spark Connector (delta-sharing-spark_2.12:0.14.0 via %%configure spark.jars.packages)
Delta Sharing Python client (delta-sharing PyPI package, v1.1+)
REST + spark.read.parquet() fallback via pre‑signed file URLs

All of these approaches work for simple Delta tables, but they fail on tables that use the Deletion Vectors feature (delta.enableDeletionVectors = true).

3. Error Messages:

REST Read → HTTP 400 with DS_UNSUPPORTED_DELTA_TABLE_FEATURES: Table features delta.enableDeletionVectors are found in table version: …
Delta Sharing Connector → session crashes with ClassNotFound for deltaSharing format unless a JAR is attached (which Fabric does not allow)
Direct Spark → unable to interpret deletion vectors, causing read failures

4. Fabric Limitations Identified While Trying to Resolve this Issue:

Cannot attach external JARs or use spark.jars.packages in Lakehouse Notebooks reliably
Does not support Delta Sharing REST’s "responseFormat":"delta" mode, which is required for Deletion Vectors
Spark runtime in Fabric Lakehouse cannot interpret Deletion Vectors metadata in Parquet files

5. Why I Can’t Work Around on Myr Side:

The provider of the Delta Share is unable/unwilling to disable Deletion Vectors or rewrite tables.
Fabric’s current Spark runtime lacks the downstream capability to read DV‑enabled tables.
We have no administrative access to modify the platform’s classpath or install custom connectors.

Has anyone encountered this issue. If so, how did you resolve it? Thanks!

Re: Issue Ingesting Delta Sharing Tables with Deletion Vectors in Fabric Lakehouse Notebooks

BhaveshPatel — Sat, 19 Jul 2025 08:39:55 GMT

Try single delta table at a time. It is working in my scenario. Deletion Vectors..It is more advanced version of OPTIMIZE ( Delta Lake Deletion Vectors | Delta Lake ). Also, Try creating a delta table first...it becomes easy then..

Re: Issue Ingesting Delta Sharing Tables with Deletion Vectors in Fabric Lakehouse Notebooks

lbendlin — Sat, 19 Jul 2025 00:05:57 GMT

If this is important to you please consider voting for an existing idea or raising a new one at https://ideas.fabric.microsoft.com

Re: Issue Ingesting Delta Sharing Tables with Deletion Vectors in Fabric Lakehouse Notebooks

libpekin — Sat, 19 Jul 2025 20:13:12 GMT

@BhaveshPatel

Thank you for your response. Just to clarify—the question isn’t about configuring Deletion Vectors in Fabric. Rather, we’re seeking viable workarounds within Fabric when ingesting Delta Share tables that we don’t control, particularly when the provider has enabled Deletion Vectors and is unwilling to modify the table properties.

What strategies exist to read or ingest from such sources given current Fabric platform limitations?

Re: Issue Ingesting Delta Sharing Tables with Deletion Vectors in Fabric Lakehouse Notebooks

libpekin — Sat, 19 Jul 2025 20:27:59 GMT

This post has been added to Fabric Ideas. @lbendlin @BhaveshPatel please consider voting for this idea. Thanks!

Re: Issue Ingesting Delta Sharing Tables with Deletion Vectors in Fabric Lakehouse Notebooks

BhaveshPatel — Sun, 20 Jul 2025 06:28:57 GMT

Then ask your fabric admins to enable Deletion Vector table properties. Use OPTIMIZE TABLE first if possible..

Re: Issue Ingesting Delta Sharing Tables with Deletion Vectors in Fabric Lakehouse Notebooks

v-menakakota — Mon, 21 Jul 2025 05:33:11 GMT

Hi @libpekin ,

Thanks for reaching out to the Microsoft fabric community forum. can you please share the link of the post here.

Thank you.

Re: Issue Ingesting Delta Sharing Tables with Deletion Vectors in Fabric Lakehouse Notebooks

v-menakakota — Fri, 25 Jul 2025 05:43:46 GMT

Hi @libpekin ,

Thank you for sharing your idea in the Ideas forum.

We appreciate your contribution. Please note that Microsoft regularly reviews ideas shared by the community, and your suggestion may be considered for future updates.

Thank you once again, and we encourage you to continue engaging with the Microsoft Fabric Community.

Best regards,
Community Support Team.

Re: Issue Ingesting Delta Sharing Tables with Deletion Vectors in Fabric Lakehouse Notebooks

Tamanchu — Thu, 26 Mar 2026 19:49:27 GMT

Hi,

This is a known limitation in the current Fabric Spark runtime, it doesn't natively support reading Delta Sharing tables that have Deletion Vectors enabled. Here's why and what you can do about it.

Why it fails:

The Fabric Spark runtime cannot interpret Deletion Vectors metadata embedded in Parquet files. Delta Sharing's REST API also doesn't support the responseFormat: delta mode required for Deletion Vectors. And unlike Databricks, Fabric Lakehouse notebooks don't allow attaching external JARs (spark.jars.packages is restricted).

Reference: Delta Lake table format interoperability — Microsoft Learn

Workarounds (from most to least practical):

Ask the data provider to create a Deletion Vector-free view/table If they can run REORG TABLE <table_name> APPLY (PURGE) on their side, it physically removes deletion vectors by rewriting Parquet files. This is the cleanest solution.

Use an intermediate landing zone outside Fabric

Read the Delta Sharing table from an Azure Databricks notebook (which supports Deletion Vectors natively)
Write the result as clean Parquet/Delta to an ADLS Gen2 location
Use a Fabric Shortcut or Copy Activity to bring it into your Lakehouse
# In Databricks notebook
df = spark.read.format("deltaSharing").load("<profile>#<share>.<schema>.<table>")
df.write.format("delta").mode("overwrite").save("abfss://<container>@<storage>.dfs.core.windows.net/clean_landing/")
Then in Fabric, create a OneLake shortcut pointing to that ADLS Gen2 path.

Use Fabric Data Pipeline with Copy Activity Instead of notebooks, use a Pipeline Copy Activity with the source configured as the external storage where the provider can export data. This bypasses the Spark runtime entirely.

Submit a feature request on Fabric Ideas (which you already did — great move!) The Fabric team is actively working on expanding Delta Lake feature support. Deletion Vectors support in the Spark runtime is one of the most requested features.

Reference: Delta Table Maintenance in Microsoft Fabric

Hope this helps! Give a kudos. Let me know if you need more details on the intermediate landing zone approach.