Support delta change feed on Data Warehouse

[simon_wicks]

It would be really useful to enable change history on data warehouse tables in Fabric as we have a CDC process to move data between two warehouses. If we could enable this like we can on Lakehouses or in Databricks we can run a targeted CDC process

https://learn.microsoft.com/en-us/azure/databricks/delta/delta-change-data-feed

[fbcideas_migusr]

Would also like to have this

[matthew_mills]

1) The ability to set Spark Table Properties on the underlying Delta Table that is managed by the Warehouse would potentially enable this and many other features.

2) Fabric Mirroring is especially important to have this feature. Fabric is using CDC to replicate a data source into the lake - but we can't use CDC to get that data out of the mirrored database, even though its all Delta under the hood.

[rhys_lewis]

I also had this issue,

I am currently working to enable Change Data Feed (CDF) on a Delta table that is sourced from a mirrored Azure SQL database. However, I have encountered an issue when trying to perform this operation within a notebook. The error message, OperationNotAllowedOnTheShortcutPath, indicates that there are restrictions on enabling Change Data Feed for a database that has been shortcutted into a lakehouse.

Given this restriction, it appears that the only viable solution is to enable Change Data Feed through the SQL endpoint in the warehouse.

However, the SET TBLPROPERTIES command is not supported in the SQL endpoint, which presents a challenge.

[cbaldorjr]

Hi Fabric Team,

Adding Delta Change Data Feed support particularly in the mirrored Azure SQL scenario would allow for ELT scenarios to be accommodated more efficiently. The pattern we are looking to implement is based on the Fabric blog below:

Bridging Fabric Lakehouses: Delta Change Data Feed for Seamless ETL (dated March 18, 2024 by Marc Bushong)

The lack of Delta Change Data Feed is particularly a challenge given that rowversion/timestamps are not supported.