topic Re: Incremental Data Load Options in Microsoft Fabric in Data Engineering

Incremental Data Load Options in Microsoft Fabric

jashfabric — Fri, 22 Mar 2024 08:20:41 GMT

Hi everyone,
I'm building a Medallion Architecture in Microsoft Fabric and am exploring options for implementing incremental data loads. Does Fabric offer built-in functionalities to achieve this? If so, what are the available approaches (e.g., Data Factory pipelines, Spark notebooks)?

Any insights or best practices for implementing incremental loads in Fabric would be highly valuable.
Thanks!
Tagging Anonymous

Re: Incremental Data Load Options in Microsoft Fabric

Anonymous — Fri, 22 Mar 2024 08:44:18 GMT

Hi jashfabric
Thanks for using Fabric Community.
You can use a Dataflow gen2 for incremental load. Incrementally amassing data in a data destination requires a technique to load only new or updated data into your data destination. This technique can be done by using a query to filter the data based on the data destination.

For more information please refer to this link:
https://learn.microsoft.com/en-us/fabric/data-factory/tutorial-setup-incremental-refresh-with-dataflows-gen2

Hope this helps. Please let me know if you have any further queries.

Re: Incremental Data Load Options in Microsoft Fabric

AndyDDC — Fri, 22 Mar 2024 10:38:29 GMT

Hi jashfabric from a Spark/Notebook/Delta perspective, you could look into using Change Data Feed:

Delta Lake Change Data Feed in Fabric Lakehouses (serverlesssql.com)

Re: Incremental Data Load Options in Microsoft Fabric

jashfabric — Fri, 22 Mar 2024 11:59:36 GMT

Thanks for your help Anonymous AndyDDC
That really helps.

Re: Incremental Data Load Options in Microsoft Fabric

Anonymous — Fri, 22 Mar 2024 12:02:33 GMT

Hi jashfabric
Glad that your query got resolved. Please continue using Fabric Community for any help regarding your queries.

Re: Incremental Data Load Options in Microsoft Fabric

jovianaditya — Mon, 25 Mar 2024 02:37:58 GMT

Hello, i want to ask some question. Is this method (from the link that you provided) is only works for append condition? If i have 100 rows from day one, and for the next day there are 2 rows that have been modified will that method also automatically update my 2 rows?

Re: Incremental Data Load Options in Microsoft Fabric

NotebookEnjoyer — Mon, 17 Nov 2025 15:36:03 GMT

I think there is no "merge/update" write option in Dataflows. You can only overwrite the whole table or append. It is also not uncommon to keep the old versions of rows as well, for logging purposes.