Reduce increment size for increment refresh in dataflows

[magnus_b]

Hi!

I have set up a dataflow that fetches data from an API using parameters similar to this: https://api.example.com/getData?fromDate=2021-09-08&toDate=2021-09-09

I have enabled incremental refresh, but the API request will fail since Power BI is trying to fetch 3 months of data at a time. Is there a way I can control the increment size to only fetching 1 month of data (or less) at a time?

When doing the API request manually in Postman I am able to run it successfully for 1 month of data, but 3 months of data will fail with error "Export result too large, please limit export criteria".

[TomMartens]

Hey @magnus_b ,

please check if this article provides what you are looking for: https://docs.microsoft.com/en-us/power-query/dataflows/incremental-refresh

If not, please provide detailed information on what makes your requirement special that is not met by the article.

Regards,

Tom

[magnus_b]

@TomMartens I have read the article, but I can't find any way to control the size of partions/increments use during inital load of data, can you?

[TomMartens]

Hey @magnus_b ,

from my knowledge it's very difficult to get the initial load done using dataflows, if the dataflows are not running on a Premium Capacity. Here you can use the enhanced compute engine (a SQL Server engine) that allows query folding.

Unfortunately, I have nothing that I can't share where the initial load is split into smaller chunks before the incremental refresh kicks in.

Regards,

Tom