Forum Discussion

jashfabric's avatar
jashfabric
Helper II
2 years ago
Solved

Incremental Data Load Options in Microsoft Fabric

Hi everyone, I'm building a Medallion Architecture in Microsoft Fabric and am exploring options for implementing incremental data loads. Does Fabric offer built-in functionalities to achieve this? I...
data engineering
Anonymous's avatar
Anonymous
Not applicable
2 years ago

Hi jashfabric 
Thanks for using Fabric Community.
You can use a Dataflow gen2 for incremental load. Incrementally amassing data in a data destination requires a technique to load only new or updated data into your data destination. This technique can be done by using a query to filter the data based on the data destination.

For more information please refer to this link:
https://learn.microsoft.com/en-us/fabric/data-factory/tutorial-setup-incremental-refresh-with-dataflows-gen2

Hope this helps. Please let me know if you have any further queries.

  • jashfabric's avatar
    jashfabric
    Helper II
    2 years ago

    Thanks for your help Anonymous  AndyDDC 
    That really helps.

    • Anonymous's avatar
      Anonymous
      Not applicable
      2 years ago

      Hi jashfabric 
      Glad that your query got resolved. Please continue using Fabric Community for any help regarding your queries.

  • jovianaditya's avatar
    jovianaditya
    Frequent Visitor
    2 years ago

    Hello, i want to ask some question. Is this method (from the link that you provided) is only works for append condition? If i have 100 rows from day one, and for the next day there are 2 rows that have been modified will that method also automatically update my 2 rows?

    • NotebookEnjoyer's avatar
      NotebookEnjoyer
      Advocate II
      9 months ago

      I think there is no "merge/update" write option in Dataflows. You can only overwrite the whole table or append. It is also not uncommon to keep the old versions of rows as well, for logging purposes.