Forum Discussion

magnus_b's avatar
magnus_b
Icon for Advocate II rankAdvocate II
5 years ago

Reduce increment size for increment refresh in dataflows

Hi!   I have set up a dataflow that fetches data from an API using parameters similar to this: https://api.example.com/getData?fromDate=2021-09-08&toDate=2021-09-09   I have enabled incremental r...
Dataflow
Need Help
magnus_b's avatar
magnus_b
Icon for Advocate II rankAdvocate II
5 years ago

TomMartens I have read the article, but I can't find any way to control the size of partions/increments use during inital load of data, can you?

TomMartens's avatar
TomMartens
Icon for Super User rankSuper User
5 years ago

Hey magnus_b ,

 

from my knowledge it's very difficult to get the initial load done using dataflows, if the dataflows are not running on a Premium Capacity. Here you can use the enhanced compute engine (a SQL Server engine) that allows query folding.

Unfortunately, I have nothing that I can't share where the initial load is split into smaller chunks before the incremental refresh kicks in.

 

Regards,

Tom