Forum Discussion

shuhn1229's avatar
shuhn1229
Resolver I
2 years ago
Solved

Daisy-Chaining Dataflows

Hi all,   I am working with an API that is very finnicky and is prone to timing out, especially when large amounts of data are requested. Unfortunately, for this use case I have to move about 10M r...
Need Help
  • shuhn1229's avatar
    shuhn1229
    2 years ago

    solution for this - multiple api calls of a reasonable size, partioning the data. 

     

    setup a power automate trigger, when a data flow completes - this will trigger the next flow with the next chunk of data.

     

    setup a final PQ step to append the query results, reference the tables from the partioned API calls. importantly, ensure enable dataload is disabled with all of the query results. append the results.

shuhn1229's avatar
shuhn1229
Resolver I
2 years ago

solution for this - multiple api calls of a reasonable size, partioning the data. 

 

setup a power automate trigger, when a data flow completes - this will trigger the next flow with the next chunk of data.

 

setup a final PQ step to append the query results, reference the tables from the partioned API calls. importantly, ensure enable dataload is disabled with all of the query results. append the results.