Daisy-Chaining Dataflows

[shuhn1229]

Hi all,

I am working with an API that is very finnicky and is prone to timing out, especially when large amounts of data are requested. Unfortunately, for this use case I have to move about 10M rows, and paginating of one big request does not help speed up the call - still results in timeouts. To get around this I basically setup ~10 data flows that act in concert. Dataflow 1 performs a call of 1/10th the data, then dataflow 2 references dataflow 1, makes a call for the next chunk of data, appends it to the referenced dataflow 1 and so on. What I have found is that on refresh of dataflow 1, all of the linked tables across the 10 flows also refresh, however the new api call doesn't refresh. This whole process seems awfully inefficient. Is there an ideal way of doing this? I thought about using PA to refresh a dataflow after one is complete trigger, going through all ten. I do not want to combine all 10 into one flow because I am very confident this will lead to timeouts.

Thanks!

[shuhn1229 · Accepted solution]

solution for this - multiple api calls of a reasonable size, partioning the data. 

setup a power automate trigger, when a data flow completes - this will trigger the next flow with the next chunk of data.

setup a final PQ step to append the query results, reference the tables from the partioned API calls. importantly, ensure enable dataload is disabled with all of the query results. append the results.

[vanessafvg]

what do you mean PA?  do you have dedicated / premium capacity, i dont think you can reference dataflows from each other without it. 

have you should tried combine the dataflows in a power be semantic model?    not quite clear what you saying, if you create 10 dataflows in sequence but do no referencing does that help?