topic Please tell me how to incrementally load data in Fabric. in Pipelines

Please tell me how to incrementally load data in Fabric.

minseok_kr — Wed, 24 Jul 2024 01:59:22 GMT

hello. I am trying to load data into Dynamics 365 using Fabric in Korea.

Please understand even if my English is not good.

I am trying to load tables existing in database A into Lakehouse.

The value of the parameter is set to yesterday's date as default, and if new data was created based on yesterday's date, it is Insert.

Or, if the data has been updated, try to update the data. In other words, we want to load incrementally based on date.

Secondly, I want to insert and update data between two specified dates. How should I configure this in Fabric?

I'm new to Fabric, so setting up this process is very challenging for me.

Re: Please tell me how to incrementally load data in Fabric.

frithjof_v — Wed, 24 Jul 2024 07:00:48 GMT

Assuming you want to use Dynamics as the source and Fabric Lakehouse as the destination:

If you want to just append new data, there is a pattern for that in Dataflow Gen2: https://learn.microsoft.com/en-us/fabric/data-factory/tutorial-setup-incremental-refresh-with-dataflows-gen2

You can probably create something similar in Data pipeline also, if you are able to filter the data based on a date parameter.

However, if you want to both append new data and update existing data (aka upsert) then I think you will need to use Notebook. There is some code examples for that here: https://microsoftlearning.github.io/mslearn-fabric/Instructions/Labs/03b-medallion-lakehouse.html

Re: Please tell me how to incrementally load data in Fabric.

frithjof_v — Wed, 24 Jul 2024 06:59:57 GMT

Please vote here to get upsert functionality in Data pipeline:

https://ideas.fabric.microsoft.com/ideas/idea/?ideaid=87f3d168-6022-ee11-a81c-6045bdc01ce4

Please vote here to get incremental refresh functionality in Dataflow Gen2:

https://ideas.fabric.microsoft.com/ideas/idea/?ideaid=4814b098-efff-ed11-a81c-6045bdb98602

Re: Please tell me how to incrementally load data in Fabric.

frithjof_v — Wed, 24 Jul 2024 07:05:30 GMT

If I misunderstood the question, and you want to upsert data into Dataverse (i.e. your destination is Dataverse), maybe you can get inspiration from this video:

https://youtu.be/WJCP1TwXBYc?si=JD9ev2L7opQb4qsh

https://learn.microsoft.com/en-us/fabric/data-factory/connector-dataverse-copy-activity

If you want to upsert into Dynamics (i.e. Dynamics is your destination), you can have a look at this documentation:

https://learn.microsoft.com/en-us/fabric/data-factory/connector-dynamics-crm-copy-activity

Re: Please tell me how to incrementally load data in Fabric.

Anonymous — Thu, 25 Jul 2024 01:29:50 GMT

Hi minseok_kr ,

Your solution is great, frithjof_v . It worked like a charm! Here I have another idea in mind, and I would like to share it for reference.

I think you can read this document for a study: Incrementally load data from Data Warehouse to Lakehouse - Microsoft Fabric | Microsoft Learn and Pattern to incrementally amass data with Dataflow Gen2 - Microsoft Fabric | Microsoft Learn

Then you can change data type.

Finally you can see what you want.

Best Regards

Yilong Zhou

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

Re: Please tell me how to incrementally load data in Fabric.

Anonymous — Tue, 30 Jul 2024 09:53:42 GMT

Can you check this doc, though it's not Dataverse as source, but similar approach: https://learn.microsoft.com/en-us/fabric/data-factory/tutorial-incremental-copy-data-warehouse-lakehouse