Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

The ultimate Microsoft Fabric, Power BI, Azure AI & SQL learning event! Join us in Las Vegas from March 26-28, 2024. Use code MSCUST for a $100 discount. Register Now

Reply
teodorajuduc
Regular Visitor

Power BI Incremental Refresh fails with Too Many Requests

Hi,

 

I'm trying to use the Incremental Refresh functionality in PBI Dataflows by connecting to 3 different tables via ODATA feed. All these tables are part of the same dataflow.

If I refresh the dataflow without using Incremental refresh, everything is refreshed in about 18 minutes. I'm trying to improve the performance as much as I can, so I enabled the Incremental Refresh. The initial refresh now fails with Too Many Request error message.

Just for test purposes, I split each table in a separate dataflow, and I noticed that one of them successfully completed, but the other 2 still fail. Below are the tables and # of rows.

Even though there is no big difference between Table 2 and Tables 3, Table 2 fails all the time when trying to read the same batch of data.

 

Entities

# of Rows

Incremental refresh status

Table 1

902,960

429 – too many requests

Table 2

202,130

429 – too many requests

Table 3

171,890

Completed in about 5 seconds

 

Since the refresh works when not using incremental refresh, do you know what can cause getting Too Many Requests error message? Do you know what's the mechanism behind when using and when not using incremental refresh?

 

Thanks,

Teodora

4 REPLIES 4
jeffshieldsdev
Solution Sage
Solution Sage

What do you have for store and refresh parameters? Incremental refresh will make a call for each date granularity in the refresh range--so if you have 10 days set, it'll make 10 calls, if you have 2 months, it'll make two--if you have 60 days, it'll make 60 calls. Theses may be executing concurrently.

 

can you move the queries to separate dataflows?

Hi Jeff,

 

I moved each table in a separate dataflow and noticed that one of them successfully completed. Even though there is no big size difference between Tables 2 and Table 3, Table 2 fails.

 

The parameters I currently set are to bring data from the last 10 years and the granularity is 1 day. I think that might be one of the issues, at least for Table 1. I will give it a try to store data only for the last 2-3 years and the granularity to be weekly. Will come back with the end result, thanks for your help.

In my example here, initial refresh will query last 5 full years (so starting 2015-01-01), each refresh afterwards will clear out the previous 3 days and make three calls to refresh for each day.

jeffshieldsdev_0-1622734019306.png

DateTime column should be the create date for a record--don't use modified date.  Incremental refresh is for appending new records, the only records updated will be those in the "refresh rows" range (since that data is cleared out and requeried).

I'm using Creation Date, I know that if I use Update Date I will start having duplicates.

This is the current configuration, the initital refresh is the one that fails. If I don't use incremental refresh, the initial refresh completes in about 18 minutes.

 

teodorajuduc_0-1622735553807.png

 

 

Helpful resources

Announcements
Fabric Community Conference

Microsoft Fabric Community Conference

Join us at our first-ever Microsoft Fabric Community Conference, March 26-28, 2024 in Las Vegas with 100+ sessions by community experts and Microsoft engineering.

February 2024 Update Carousel

Power BI Monthly Update - February 2024

Check out the February 2024 Power BI update to learn about new features.

Fabric Career Hub

Microsoft Fabric Career Hub

Explore career paths and learn resources in Fabric.

Fabric Partner Community

Microsoft Fabric Partner Community

Engage with the Fabric engineering team, hear of product updates, business opportunities, and resources in the Fabric Partner Community.