Experimenting with Copy Job. Yeah, OK, but not great. Some suggestions:
1) Make the option for OVERWRITE available for Warehouse destinations. Currently available for Lakehouses only. This article ...
(this one is listed under a separate Idea:)
3) Allow the Copy Job INCREMENTAL to use a TIMESTAMP or ROWVERSION SQL data type as the watermark column. This data type column is ABSOLUTELY the best type for watermarks. It is: a) immutable, b) non-null always, c) system managed, and d) immune to updates from user processes, and finally, e) managed by the SQL engine (no triggers or app code required to keep it updated).
The Azure Data Factory Metadata Driven Copy Wizard is a precursor to the Fabric Copy Job and even THAT did not support TIMESTAMP or ROWVERSION types as watermark. But I have been able to hack those pipelines and Control JSON to make it work.
Think about it, this is what this datatype was BORN to do! And conceived and delivered by Microsoft in the SQL engine 20 or so years ago. Yet the Fabric people seem to shun it. So disappointing!
