Forum Discussion
Incremental refresh using Fabric pipeline
Hi Everyone,
I am looking for a way/documentation I can uset to build a incremental refresh using pipelines within Microsoft Fabric.
My data source is on-prem postgres. Pipeline recently started supporting on-prem sources in MS fabric - How to copy data using copy activity - Microsoft Fabric | Microsoft Learn
however I am not sure on how I can build a incremental refresh using pipelines. could you please help to guide/provide any helpful documentation on it.
6 Replies
- NandanHegde
Super User
You can use the concept of watermark
The below MSFT doc :
https://learn.microsoft.com/en-us/azure/data-factory/tutorial-incremental-copy-overview
provides in detail for incremental load
NandanHegde Anonymous - can this be used within fabric pipeline for on-prem postgres source the document refers to ADF and Synapse only ?
- NandanHegde
Yes,
The core concept remains the same irrespective of whatever the source is.
Also fabric data pipelines are saas replica of Azure data factory
Hi Chhayav12 ,
Thanks for using Fabric Community.
Incremental refresh is listed on the public roadmap.
Docs to refer - What's new and planned for Data Factory in Microsoft Fabric - Microsoft Fabric | Microsoft Learn
As NandanHegde already shared the document.
Adding on to it, you can also refer this document, inorder to achive incremental load-
CHANGE (IN THE HOUSE OF LAKES) - It's Not About The Cell (itsnotaboutthecell.com)
Hope this is helpful. Please let me know incase of further queries.Hi Chhayav12 ,
We haven’t heard from you on the last response and was just checking back to see if your query was answered.
Did you got some insights from above links?
Otherwise, will respond back with the more details and we will try to help you.Hi Chhayav12 ,
We haven’t heard from you on the last response and was just checking back to see if your query was answered.
Did you got some insights from above links?
Otherwise, will respond back with the more details and we will try to help you.
