Forum Discussion
Incremental Data Load Options in Microsoft Fabric
Hi everyone,
I'm building a Medallion Architecture in Microsoft Fabric and am exploring options for implementing incremental data loads. Does Fabric offer built-in functionalities to achieve this? If so, what are the available approaches (e.g., Data Factory pipelines, Spark notebooks)?
Any insights or best practices for implementing incremental loads in Fabric would be highly valuable.
Thanks!
Tagging Anonymous
Hi jashfabric
Thanks for using Fabric Community.
You can use a Dataflow gen2 for incremental load. Incrementally amassing data in a data destination requires a technique to load only new or updated data into your data destination. This technique can be done by using a query to filter the data based on the data destination.
For more information please refer to this link:
https://learn.microsoft.com/en-us/fabric/data-factory/tutorial-setup-incremental-refresh-with-dataflows-gen2
Hope this helps. Please let me know if you have any further queries.Hi jashfabric from a Spark/Notebook/Delta perspective, you could look into using Change Data Feed:
Delta Lake Change Data Feed in Fabric Lakehouses (serverlesssql.com)
6 Replies
Hi jashfabric from a Spark/Notebook/Delta perspective, you could look into using Change Data Feed:
Delta Lake Change Data Feed in Fabric Lakehouses (serverlesssql.com)
Hi jashfabric
Thanks for using Fabric Community.
You can use a Dataflow gen2 for incremental load. Incrementally amassing data in a data destination requires a technique to load only new or updated data into your data destination. This technique can be done by using a query to filter the data based on the data destination.
For more information please refer to this link:
https://learn.microsoft.com/en-us/fabric/data-factory/tutorial-setup-incremental-refresh-with-dataflows-gen2
Hope this helps. Please let me know if you have any further queries.Thanks for your help Anonymous AndyDDC
That really helps.Hi jashfabric
Glad that your query got resolved. Please continue using Fabric Community for any help regarding your queries.
Hello, i want to ask some question. Is this method (from the link that you provided) is only works for append condition? If i have 100 rows from day one, and for the next day there are 2 rows that have been modified will that method also automatically update my 2 rows?
I think there is no "merge/update" write option in Dataflows. You can only overwrite the whole table or append. It is also not uncommon to keep the old versions of rows as well, for logging purposes.
