topic Incremental refresh using Fabric pipeline in Pipelines

Incremental refresh using Fabric pipeline

Chhayav12 — Mon, 08 Apr 2024 08:37:00 GMT

Hi Everyone,

I am looking for a way/documentation I can uset to build a incremental refresh using pipelines within Microsoft Fabric.
My data source is on-prem postgres. Pipeline recently started supporting on-prem sources in MS fabric - How to copy data using copy activity - Microsoft Fabric | Microsoft Learn

however I am not sure on how I can build a incremental refresh using pipelines. could you please help to guide/provide any helpful documentation on it.

Re: Incremental refresh using Fabric pipeline

NandanHegde — Mon, 08 Apr 2024 09:03:27 GMT

You can use the concept of watermark

The below MSFT doc :

https://learn.microsoft.com/en-us/azure/data-factory/tutorial-incremental-copy-overview

provides in detail for incremental load

Re: Incremental refresh using Fabric pipeline

Anonymous — Mon, 08 Apr 2024 09:42:31 GMT

Hi Chhayav12 ,

Thanks for using Fabric Community.

Incremental refresh is listed on the public roadmap.

Docs to refer - What's new and planned for Data Factory in Microsoft Fabric - Microsoft Fabric | Microsoft Learn

As NandanHegde already shared the document.
Adding on to it, you can also refer this document, inorder to achive incremental load-
CHANGE (IN THE HOUSE OF LAKES) - It's Not About The Cell (itsnotaboutthecell.com)

Hope this is helpful. Please let me know incase of further queries.

Re: Incremental refresh using Fabric pipeline

Chhayav12 — Tue, 09 Apr 2024 04:27:03 GMT

NandanHegde Anonymous - can this be used within fabric pipeline for on-prem postgres source the document refers to ADF and Synapse only ?

Re: Incremental refresh using Fabric pipeline

NandanHegde — Tue, 09 Apr 2024 05:07:36 GMT

Yes,

The core concept remains the same irrespective of whatever the source is.

Also fabric data pipelines are saas replica of Azure data factory

Re: Incremental refresh using Fabric pipeline

Anonymous — Wed, 10 Apr 2024 08:16:16 GMT

Hi Chhayav12 ,

We haven’t heard from you on the last response and was just checking back to see if your query was answered.
Did you got some insights from above links?
Otherwise, will respond back with the more details and we will try to help you.

Re: Incremental refresh using Fabric pipeline

Anonymous — Thu, 11 Apr 2024 09:07:22 GMT