Forum Discussion

wilda's avatar
wilda
New Member
4 years ago

Incremental refresh in dataflows - where to apply it?

Hi community,

(first post here! and PBI rookie)

 

I have a solution based on several dataflows. The orchestration of them is still a little bit confusing to me, but here's the specific matter that brings me here:

 

- I have a dataflow A that acts as an ingestion one, that is, reads from the data source and only applies some filters. It takes ~35 minutes to refresh.

- I have another dataflow B that reads from dataflow A (linked entities) and applies several transformations. The refresh time of this dataflow is ~4 HOURS!

 

I figured that this would need some incremental refresh, but then I thought: where to apply the incremental refresh: dataflow A or B? Or both? And is this really going to help with the refreshing times? Because the whole data reading is done in dataflow A, and that is relatively quick... And how does an incremental refresh in dataflow A pass only the new/updated rows to dataflow B? That is, how can dataflow B understand that it should run all transformations only on the new records?

 

I'm working on a Premium workspace.

 

Thanks in advance for any kind of input,

Lisa

1 Reply