Forum Discussion
Reduce increment size for increment refresh in dataflows
Hi!
I have set up a dataflow that fetches data from an API using parameters similar to this: https://api.example.com/getData?fromDate=2021-09-08&toDate=2021-09-09
I have enabled incremental refresh, but the API request will fail since Power BI is trying to fetch 3 months of data at a time. Is there a way I can control the increment size to only fetching 1 month of data (or less) at a time?
When doing the API request manually in Postman I am able to run it successfully for 1 month of data, but 3 months of data will fail with error "Export result too large, please limit export criteria".
3 Replies
Hey magnus_b ,
please check if this article provides what you are looking for: https://docs.microsoft.com/en-us/power-query/dataflows/incremental-refresh
If not, please provide detailed information on what makes your requirement special that is not met by the article.
Regards,
Tom
TomMartens I have read the article, but I can't find any way to control the size of partions/increments use during inital load of data, can you?
Hey magnus_b ,
from my knowledge it's very difficult to get the initial load done using dataflows, if the dataflows are not running on a Premium Capacity. Here you can use the enhanced compute engine (a SQL Server engine) that allows query folding.
Unfortunately, I have nothing that I can't share where the initial load is split into smaller chunks before the incremental refresh kicks in.
Regards,
Tom
