Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

Find everything you need to get certified on Fabric—skills challenges, live sessions, exam prep, role guidance, and more. Get started

Reply
jp23
Frequent Visitor

Incremental Refresh Fails When More Than 6 Months

Hello Community -

 

We have several dataflows that are setup to for incremental refresh based on various web API's with store rows from the past 1 year and refresh frows from the past 5 days. They all work, except for one specific API which doesn't seem to want to allow to go past 6 months for 'store rows from the past', it sometimes spins forever (10+ hours) and sometimes just fails with a message stating "make your dataset smaller".  Because the API limits the results at 100 rows per request on this particular dataflow, I suspect it's a timeout issue but can't really pinpoint the issue based on the refresh history logs, they are very vague. This should return around a million rows when all said and done for the first import.

 

Could this be related to our workload settings for the workspace? For example, we have the following configuration setup:

 

- Query memory Limit -> 0%

- Query Timeout -> 3600

- Max Intermdiate Row Count -> 100000

- Max Result Row Count -> 2147483647

 

Any guidance is greatly appreciated.

3 REPLIES 3
jp23
Frequent Visitor

Thanks for your response! Yes, we have contacted the vendor and they pointed back to a Power Bi timeout value. They ensured there is no timeout or rate limiting on the endpoint.

 

I've tried different variations of [Timeout=#duration(0, 0, 0, 30) with no success. After the first 6 months of data (approx ~1 hour) the following 6 months of data goes on for an endless duration until it eventually fails (sometimes 7+ hours).

sdk88
New Member

Hello @jp23 

To me it appears like if API is fast enough to provide response then next layer is consuming data in PBI. I am assuming there is lots of data and processing it as for M Query for table may be requring time.

GilbertQ
Super User
Super User

Hi @jp23 

 

What it could be is that the API you are calling has got a timeout limit too, and because you have to call it 100 rows at a time it could be the issue.


Because it is working for other APIs, there is certainly no issue with what you have done or the PBI settings. I would look to contact this particular API and see if there is anyway to overcome this?





Did I answer your question? Mark my post as a solution!

Proud to be a Super User!







Power BI Blog

Helpful resources

Announcements
July 2024 Power BI Update

Power BI Monthly Update - July 2024

Check out the July 2024 Power BI update to learn about new features.

July Newsletter

Fabric Community Update - July 2024

Find out what's new and trending in the Fabric Community.