dynamic refresh of data

[arkiboys2]

hello,
The data factory pipeline loads data from sql server into the warehouse tables.
This is done by dropping the warehouse table before the load.

Question:

How can I do incremental refresh into the table in the warehouse?

thanks

[NandanHegde · Accepted solution]

You can follow the below process :

Drop and create a staging table with the delta data from source

then write a custom logic to upsert /incremental load into destination table( as merge is not yet supported in fabric warehouse)

[v-shex-msft · Accepted solution]

Hi @arkiboys2,

You can refer to the following tutorial to 'add the copy activity to copy incremental data' in data pipeline.(you need to modify the output destination to data warehouse instead of lakehouse)

Incrementally load data from Data Warehouse to Lakehouse - Microsoft Fabric | Microsoft Learn

Regards,

Xiaoxin Sheng

[NandanHegde · Accepted solution]

You can follow the below process :

Drop and create a staging table with the delta data from source

then write a custom logic to upsert /incremental load into destination table( as merge is not yet supported in fabric warehouse)