Can aggressive retry on OData requests be reduced?

[monty]

We provide a proxy for 75+ (cloud)platforms using OData4. This enables Power BI, Power Query and ADF to easily exchange data with all these platforms.

However, when an error occurs, Power BI tries each table in total approximately 10 times, typically all without success. With a report with 10 tables, 100 OData4 requests are executed.

For instance, we have made a OData4 source unavailable such that it raises a permanent error (see picture). On Power BI Desktop with data preview, it first tries a number of times WITH $top=1000 and then more WITHOUT $top=1000.

The request and headers for the rest are all the same:

Request /invantive-exact-online/odata4/ExactOnlineDataDictionary.DataDictionary.IncrementalLoadEventLogEntries@eol

Headers:
Accept: application/json;odata.metadata=minimal;q=1.0,application/json;odata=minimalmetadata;q=0.9,application/atomsvc+xml;q=0.8,application/atom+xml;q=0.8,application/xml;q=0.7,text/plain;q=0.7
Accept-Encoding: gzip, deflate, Authorization: (header present), Host: localhost:44301
User-Agent: Microsoft.Data.Mashup (https://go.microsoft.com/fwlink/?LinkID=304225)
MaxDataServiceVersion: 3.0
OData-MaxVersion: 4.0

Some users flood our servers sometimes with thousands of requests and recently one specific user downloaded 184 GB Brotli-compressed data (2,4 TB) on a Saturday-morning, all with the same error due to a crash in ADF.

Is there a known way to convince Power BI that some OData4 errors are persistent and that it should backoff some time?

We have tried varying the HTTP status and the content of the standardized OData4 JSON format, but with no luck.

It would be great when the OData4 producer could say: this is a permanent error, try again tomorrow or try again in a few seconds.

[v-cazheng-msft]

@monty 

We should not have such a method. As long as Odata doesn't throttle the request from Power BI, Power BI will retry several times to complete operation and I haven't found a statement on how many times retries for each table Power BI will carry when connection fails. I will continue to look for it and sync with you here when there is any update.

Best Regards,

Community Support Team _Caiyun

[monty]

Thanks for your update.

We see it probing for version 3 and 4 multiple times (even with OData-Version header present). This is a fixed number of retries (at most 10).

We have also Power BI Service seen running the same download of 1 GB (gzip compressed network size) 184 times on one Saturday morning. Power BI Service crashed each time processing it and then retried few minutes later. Our operations locked out the customer as soon as the problem became clear.

Headers from the OData Producer are (ignore Consumer, this is from a browser):

[monty]

Might relate to https://ideas.powerbi.com/ideas/idea/?ideaid=578f015b-11e1-4257-9d33-6f3407075416

[v-cazheng-msft]

Hi @monty 

Thanks for your reply. Judging from the current situation, there should be no way to directly control the times of connection attempts from Service to the OData data source when connection fails. If you really need it, vote for the same idea in the link Microsoft Idea  · Multiple requests to the OData server (powerbi.com) is an option you can take currently.

If you need OData4 producer could say: this is a permanent error, try again tomorrow or try again in a few seconds, I think you should look for this from the data source you use. Power BI cannot control this.

Best Regards,

Community Support Team _Caiyun

[monty]

Thank you @v-cazheng-msft . The other idea is a different problem. The other one refers to the fact that Power BI tries OData 3 and 4 in various flavours to retrieve data. However, that typically triples the number of tries.

In this case, PowerBI.com keeps trying to download the data set, sometimes even hundreds of times. This effectively seems a denial-of-service attack. It re-appears every week or so and from the end user point of view they think something is wrong with our OData producer. We can then show to them that the data is actually completely delivered and PowerBI.com seems to crash internally or somehow else needing to download the dataset again.

The suggested solution is to limit the maximum number of tries, just like with TCP for instance. Additionally, it would definitely be a big plus when PowerBI.com would provide meaningful insights using some kind of log of it's download activity, just like Azure Data Factory or 3rd party products.