Hi,
It would be amazing if I can set up a copy job against a CDF enabled Lakehouse delta tables and incrementally grab all Inserts Updates Deletes with Append option to destination Lakehouse or Warehouse.
For example, Link to Fabric Lakehouse against Dataverse behind Dynamic 365 as a source. It would be amazing if we can setup a Copy Job that does full load first and captures the version of the CDF table (Decribe History) and then does incremental append only loads grabbing only version changes from latest version onwards, instead of latest change.
This way, we will be able to have a immutable, append-only, read-only Bronze lakehouse that contains all changes from a source and using audit columns to have previous CDF version as lower bound and current maximum as upper bound (similar to watermark based incremental append load) system as a foundational layer for a robust SCD2 (bitemporal) Silver Layer.
Thanks!
Niko
