Incremental Refresh Fails When More Than 6 Months

[jp23]

Hello Community -

We have several dataflows that are setup to for incremental refresh based on various web API's with store rows from the past 1 year and refresh frows from the past 5 days. They all work, except for one specific API which doesn't seem to want to allow to go past 6 months for 'store rows from the past', it sometimes spins forever (10+ hours) and sometimes just fails with a message stating "make your dataset smaller".  Because the API limits the results at 100 rows per request on this particular dataflow, I suspect it's a timeout issue but can't really pinpoint the issue based on the refresh history logs, they are very vague. This should return around a million rows when all said and done for the first import.

Could this be related to our workload settings for the workspace? For example, we have the following configuration setup:

- Query memory Limit -> 0%

- Query Timeout -> 3600

- Max Intermdiate Row Count -> 100000

- Max Result Row Count -> 2147483647

Any guidance is greatly appreciated.

[jp23]

Thanks for your response! Yes, we have contacted the vendor and they pointed back to a Power Bi timeout value. They ensured there is no timeout or rate limiting on the endpoint.

I've tried different variations of [Timeout=#duration(0, 0, 0, 30) with no success. After the first 6 months of data (approx ~1 hour) the following 6 months of data goes on for an endless duration until it eventually fails (sometimes 7+ hours).

[sdk88]

Hello @jp23 

To me it appears like if API is fast enough to provide response then next layer is consuming data in PBI. I am assuming there is lots of data and processing it as for M Query for table may be requring time.

[GilbertQ]

Hi @jp23 

What it could be is that the API you are calling has got a timeout limit too, and because you have to call it 100 rows at a time it could be the issue.

Because it is working for other APIs, there is certainly no issue with what you have done or the PBI settings. I would look to contact this particular API and see if there is anyway to overcome this?