topic Re: dynamic refresh of data in Data Engineering

dynamic refresh of data

arkiboys2 — Sat, 13 Jul 2024 17:37:55 GMT

hello,
The data factory pipeline loads data from sql server into the warehouse tables.
This is done by dropping the warehouse table before the load.

Question:

How can I do incremental refresh into the table in the warehouse?

thanks

Re: dynamic refresh of data

NandanHegde — Mon, 15 Jul 2024 03:43:47 GMT

You can follow the below process :

Drop and create a staging table with the delta data from source

then write a custom logic to upsert /incremental load into destination table( as merge is not yet supported in fabric warehouse)

Re: dynamic refresh of data

Anonymous — Mon, 15 Jul 2024 05:17:09 GMT

Hi @arkiboys2,

You can refer to the following tutorial to 'add the copy activity to copy incremental data' in data pipeline.(you need to modify the output destination to data warehouse instead of lakehouse)

Incrementally load data from Data Warehouse to Lakehouse - Microsoft Fabric | Microsoft Learn

Regards,

Xiaoxin Sheng