Spark Connector for MS Fabric Warehouse

Why Warehouses Are Slower
1. ACID Compliance Overhead
• Full transaction support requires locking and delayed write visibility
• Multi-table transactions add coordination costs not present in Lakehouse’s append-only writes
2. T-SQL Translation Layer
• Spark-to-Warehouse writes go through a SQL translation layer, unlike Lakehouse’s direct Parquet writes
• Adds ~40% latency compared to ABFS path writes
3. File Management
• Warehouses use smaller file sizes (avg 8MB vs 128MB in Lakehouse), increasing metadata ops
• More files per write operation exacerbate Fabric’s file count limits

No Native Spark DELETE Support
Warehouse tables require T-SQL transactions for deletions, unlike Lakehouse tables that support Delta Lake’s `DELETE` via Spark-SQL

# Requires db_writer permissions
spark.sql("""
EXECUTE SQL.warehouse.sales_db
"DELETE FROM orders WHERE ProductKey = 5"
""")

Hi fabconmvp,

Thank you for reaching out to the Microsoft Fabric Forum Community.

Here are the links to Microsoft's official documentation that address your concerns about the Spark Connector for Fabric Warehouse and its performance:
Please refer to the links below:

Warehouse performance guidelines - Microsoft Fabric | Microsoft Learn

Spark connector for Microsoft Fabric Data Warehouse - Microsoft Fabric | Microsoft Learn

If this post helps, then please give us ‘Kudos’ and consider Accept it as a solution to help the other members find it more quickly. 

Thank you.

Hi nilendraFabric can you share where you got the info "Adds ~40% latency compared to ABFS path writes" from?  The way that data is loaded from Spark into a Warehouse is it's staged first in storage then uses the fast COPY INTO process.