Forum Discussion
dynamic refresh of data
hello,
The data factory pipeline loads data from sql server into the warehouse tables.
This is done by dropping the warehouse table before the load.
Question:
How can I do incremental refresh into the table in the warehouse?
thanks
You can follow the below process :
Drop and create a staging table with the delta data from source
then write a custom logic to upsert /incremental load into destination table( as merge is not yet supported in fabric warehouse)
Hi arkiboys2,
You can refer to the following tutorial to 'add the copy activity to copy incremental data' in data pipeline.(you need to modify the output destination to data warehouse instead of lakehouse)
Incrementally load data from Data Warehouse to Lakehouse - Microsoft Fabric | Microsoft Learn
Regards,
Xiaoxin Sheng
2 Replies
You can follow the below process :
Drop and create a staging table with the delta data from source
then write a custom logic to upsert /incremental load into destination table( as merge is not yet supported in fabric warehouse)
Hi arkiboys2,
You can refer to the following tutorial to 'add the copy activity to copy incremental data' in data pipeline.(you need to modify the output destination to data warehouse instead of lakehouse)
Incrementally load data from Data Warehouse to Lakehouse - Microsoft Fabric | Microsoft Learn
Regards,
Xiaoxin Sheng