Forum Discussion
Advocate IDoes incremental refresh do anything if I don't have a data destination ?
Hi mrozzano ,
Incremental Refresh requires a data target to operate effectively. Without a data destination, the Incremental Refresh setting will have no place to store the refreshed data, rendering the setting invalid.
In your scenario, Dataflow A has incremental refresh enabled but no target, while Dataflow B gets data from Dataflow A and Dataflow B's target is a datalake repository. Due to the lack of a target to store the refreshed data, the incremental refresh in Dataflow A won't actually do anything.
For incremental refresh to work properly, you need to ensure that the Dataflow has a valid target so that the refreshed data can be stored and accessed by subsequent Dataflows or reports.For more details, you can refer to below document:
Incremental refresh in Dataflow Gen2 - Microsoft Fabric | Microsoft Learn
Best Regards,
Adamk KongIf this post helps, then please consider Accept it as the solution to help the other members find it more quickly.
Solution SageAn incremental dataflow refresh works as follows:
"
- Evaluate Changes: The dataflow compares the maximum value in the change detection column with the previous refresh. If the value has changed, the bucket is marked for processing.
- Retrieve Data: The dataflow retrieves data for the changed buckets in parallel, loading it into the staging area.
- Replace Data: The dataflow replaces the data in the destination with the new data, ensuring only the updated buckets are affected. Any historical data or data that is outside the range of buckets marked for processing is not touched or changed. This way you can retain long term history in your destination.
If you don't have a data destination, it cannot replace the updated values. I have not tested this, but I suspect that you will get the rows that were changed inserted into your second dataflow destination. However, those inserted values could be "duplicates" (with a change in at least the column that you've selected to check for changes per bucket) from previous loads. It will not replace anything. Therefore, your lakehouse will eventually contain more rows that the original source and it will keep growing when time pass by.