Support Lakehouse as Destination for Incremental Copy in Copy Job

[scca]

Not sure about others but lakehouse seems to be the go-to as my destination for most if not all workloads. Wouldn't it be lovely to just land everything in lakehouse as much as possible? That said my existing copy activities do move data from sql server into lakehouse so if copy job's incremental copy can do that, that'd be amazing. 

[yexu]

Today you can do incremental copy from source, and append or upsert to lakehouse.  

If you want to do CDC replciation to lakehouse, it was planned. 

[hedeencharles]

This would be a fantastic feature. Would this support the "merge" update method feature too that is currently in preview? As of now, I get the below error when trying to incrementally load and merge from Azure SQL DB to Fabric Lakehouse. Operation on target CopyJobActivityLoop failed: Activity failed because an inner activity failed; Inner activity name: CopyData_final, Error: ErrorCode=ApplyChangesNotSupported,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=ApplyChanges is not supported for the copy pair from AzureSqlDatabase to LakehouseTable.,Source=Microsoft.DataTransfer.ClientLibrary,'

[yexu]

Could you please try it again now to see if the issue still there?

[hedeencharles]

Sorry I was OOO for a while. This issue was resolved and I can now incrementally merge to a lakehouse table. Thanks @yexu!

[yexu]

Expanded CDC Support for More Sources & Destinations – Simplifying Data Ingestion with Copy job | Mi...