Incremental refresh using Fabric pipeline

[Chhayav12]

Hi Everyone,

I am looking for a way/documentation I can uset to build a incremental refresh using pipelines within Microsoft Fabric.
My data source is on-prem postgres. Pipeline recently started supporting on-prem sources in MS fabric - How to copy data using copy activity - Microsoft Fabric | Microsoft Learn

however I am not sure on how I can build a incremental refresh using pipelines. could you please help to guide/provide any helpful documentation on it.

[Anonymous]

Hi @Chhayav12 ,

Thanks for using Fabric Community.

Incremental refresh is listed on the public roadmap.

Docs to refer - What's new and planned for Data Factory in Microsoft Fabric - Microsoft Fabric | Microsoft Learn

As @NandanHegde  already shared the document.
Adding on to it, you can also refer this document, inorder to achive incremental load-
CHANGE (IN THE HOUSE OF LAKES) - It's Not About The Cell (itsnotaboutthecell.com)

Hope this is helpful. Please let me know incase of further queries.

[Anonymous]

Hi @Chhayav12 ,

We haven’t heard from you on the last response and was just checking back to see if your query was answered.
Did you got some insights from above links?
Otherwise, will respond back with the more details and we will try to help you.

[Anonymous]

Hi @Chhayav12 ,

We haven’t heard from you on the last response and was just checking back to see if your query was answered.
Did you got some insights from above links?
Otherwise, will respond back with the more details and we will try to help you.

[NandanHegde]

You can use the concept of watermark 

The below MSFT doc :

https://learn.microsoft.com/en-us/azure/data-factory/tutorial-incremental-copy-overview

provides in detail for incremental load

[Chhayav12]

@NandanHegde @Anonymous  - can this be used within fabric pipeline for on-prem postgres source the document refers to ADF and Synapse only ?

[NandanHegde]

Yes, 

The core concept remains the same irrespective of whatever the source is. 

Also fabric data pipelines are saas replica of Azure data factory