Incremental refresh in dataflows - where to apply it?

[wilda]

Hi community,

(first post here! and PBI rookie)

I have a solution based on several dataflows. The orchestration of them is still a little bit confusing to me, but here's the specific matter that brings me here:

- I have a dataflow A that acts as an ingestion one, that is, reads from the data source and only applies some filters. It takes ~35 minutes to refresh.

- I have another dataflow B that reads from dataflow A (linked entities) and applies several transformations. The refresh time of this dataflow is ~4 HOURS!

I figured that this would need some incremental refresh, but then I thought: where to apply the incremental refresh: dataflow A or B? Or both? And is this really going to help with the refreshing times? Because the whole data reading is done in dataflow A, and that is relatively quick... And how does an incremental refresh in dataflow A pass only the new/updated rows to dataflow B? That is, how can dataflow B understand that it should run all transformations only on the new records?

I'm working on a Premium workspace.

Thanks in advance for any kind of input,

Lisa

[Anonymous]

Hi  @wilda ,

It is recommended to configure incremental refresh in dataflow A, dataflow b is created by dataflow a, dataflow a is the original data source, incremental refresh is applied to dataflow A first, and then to dataflow B.

Refer to :

https://docs.microsoft.com/en-us/power-query/dataflows/incremental-refresh

https://community.powerbi.com/t5/Service/Using-Parameters-with-Dataflows/m-p/1019937

Best Regards,

Liu Yang

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.