Forum Discussion
Impactful IndividualFabric Data Factory Pipeline: Incremental load
Your table action options imply that you are using pipelines. If so, good news, we have just added a new feature that allows you to "upsert" the new rows into existing tables instead of append/overwrite only.
Its in private preview, please ping me with your workspace id, and we can enable it for you along with instructions. cc makromer
Microsoft EmployeeYour table action options imply that you are using pipelines. If so, good news, we have just added a new feature that allows you to "upsert" the new rows into existing tables instead of append/overwrite only.
Its in private preview, please ping me with your workspace id, and we can enable it for you along with instructions. cc makromer
Hi ajarora,
Thanks for providing the information! I have a question about using Data Factory pipelines to upsert data from an on-premise SQL Server incrementally. Is there a method available for doing this?
I've managed to establish a connection through our Gateway to the Server and would like to use the DF Copy activity to query data and upsert it into a Datalakehouse delta table. However, I'm not seeing the created connection in the Copy activity wizard. Is there something I might be missing?
On the other hand, I can connect to it using Dataflows Gen2, apply a filter, and then use the Datalakehouse as the destination for data in append mode. The problem is, I'm not quite sure how to query only new records without loading the entire table initially.
Do you know if there's a recommended method to achieve this? Especially now during the preview period of Fabric, I'd appreciate any guidance you might have.
Thanks in advance for your help!
- ajarora
Unfortunately currently Gateway based connections are not supported in Pipelines. We are actively working on it. Until then, if you can use dataflows to stage your onprem data to a cloud store (say SQLDb) then use copy to UPSERT it to LH table, that would be a workaround.