Embedded The request was blocked by DoSP

[MarcosSpeca]

Hi, I'm getting the error below when embedding a report:

"{"message":"Fail to load - Could not retrieve models and explorations.","detailedMessage":"The request was blocked by DoSP <ClientEndpoint=XXX.XXX.XXX.XXX:XXXX>","errorCode":"429","level":"6","technicalDetails":{"requestId":"f1da1f89-b99d-6589-f57f-a872aab3d819"}}"

Any thoughts? Seems this error startet since feb/21.

Thanks,

[MarcosSpeca]

Hi, I'm getting the error below when embedding a report:

"{"message":"Fail to load - Could not retrieve models and explorations.","detailedMessage":"The request was blocked by DoSP <ClientEndpoint=XXX.XXX.XXX.XXX:XXXX>","errorCode":"429","level":"6","technicalDetails":{"requestId":"f1da1f89-b99d-6589-f57f-a872aab3d819"}}"

Any thoughts? Seems this error startet since feb/21.

Thanks,

[dkruge]

Hi all, 

i have been following this as i have the same issue and been testing all day long..
I managed to get it working using a Azure integration runtime in West Germany or using a Intergration Runtime Self Hosted. 
Changed all the WEB requests that go towards the Power BI API all run successfully now. 🙂 
I received from support there is no ETA of when this will be resolved.

see response below
"

Explanation:

• This a known issue in Power BI we apologize for the inconvenience caused and our Product Team is still working on this fix and the ETA is not yet provided.
   
• The cause of this issue is certain IP of ADF Azure integration runtime (IR) in some regions is blocked due to Power BI’s DoS Protection mechanism.
   

As per the suggestions from our Product Team, Please try to use the Azure IR in other regions(outside of existing one) or use a self hosted IR in web activity as a workaround for this issue.
"

[Anonymous]

Hi Microsoft,

 A month ago I implemented the suggested workaround and moved my ADF to another region, from West Europe to France Central. This workaround worked till now.

Since a few days we are now getting the same issue on the France Central region.

It is not doable to move this to another region each time somebody else makes a lot of bad requests.

When will this known issue be fixed???

Regards,

Thomas

[dkruge]

Hi Thomas, 

I am not Microsoft but the solution moving regions is not the best, using self host integrated runtimes for me works perfect. 
In the reply from Microsoft when i opened a ticket in August they advised Late September a fix would be rolled out. 

I advise opening a ticket with Microsoft. 

Thanks 

Danny

[FGA]

 Hi all,

As of 4 days ago we start to receive the same error ("The request was blocked by DoSP ") from our Synapse PBI dataset refresh pipeline. At first it was just once maybe every 4 to 6 hours and now its 8 out of 10 attempts that get denied. Our pipeline ensures that with every attempt we get a new token based on our Service principal with the proper permission and it had been working well for more then a month.  The most frequent running pipeline is every 30 min. and the rest every 6 hours. The frequency is a requirement from our business users. Is there any UI or Microsoft department were we can indicate the PBI Service that our calls are trusted? We need to keep the current frequency and solve this issue. Any suggestion would be very much appreciated.

Regards, Fabian

[AlexanderTikh]

We have opened ticket with MS and answer was:

Suggestion
===========
Regarding your concern about this issue, we checked this with our power bi backend team, they indicated that the Power BI algorithm is designed due to security consideration of Denial-of-Service (DoS).
The policy is if the IP address sends more than a certain number of invalid requests within 1 ~ 5 minutes, the IP is blocked for 5 minutes. IP is only blocked when the requests cannot pass authentication.
If someone sends large number of requests with invalid token, PBI needs to query AAD and metadata store to authenticate the user. This can bring down our system or impact other users. while there are many customer share same IP by using ADF in same region. We fully understand it’s not reasonable that you was blocked by other customer in ADF, but from power bi side, we only receive the request and know which IP it’s sending from and which token it bring, if it’s a invalid token, it’s not possible to know who is sending it, the only thing we know is the IP of the request.

We have confirmed with our backend team that currently we cannot change this from power bi side about the IP of ADF due to security consideration of Denial-of-Service (DoS), today we got some update from backend team, they are actively working on improving this behavior like only throttle the invalid token.

Before that, we may suggest you to use the short-term workaround by changing ADF runtime to different region so that we can use a different IP to unblock this situation if possible.

[FGA]

Hi Alexander,

Thanks for this update.

Do you also happen to know how long it takes for the security algorithm (policy) before the IP address is cleared from black list ?

@Question2community: As there is currently no way to influence the PBI Security algorithm, we are then turning into looking deeper in our calls.
How can we validate (logs or similiar) what our Synapse pipelines calls to PBI REST API's are actually doing ?
Things like retrys, warning, invalid tokens, errors, etc. This can maybe then help us to pin point the probable cause.

To the suggestion of Microsoft, changing ADF runtime to different region, we would need to get clearence from our Security team first. Also, I'm curious as if this will unblock permanetly or just let calls through from other region but when swithcing back to current region the algorithm will again block our calls. Do you happen to know the behavior in this case?

Regards, Fabian

[Anonymous]

Hi Fabian, Alexander,

I implemented the suggested workaround on our DEV and TEST environment. It is a little bit more work then it seemed originally as you cannot 'move' an existing data factory to another region. Rather you have to recreate it completely in another region, and then apply all access policies and necessary security settings.

Luckily we did not had to recreate all our data factories in another region, 'moving' the "refresh dataset" data factory to another region was sufficient. (it 'moved' from West Europe to France Central).

The orchestration data factory (residing in West Europe) started the 'refresh dataset' data factory (residing in France Central) using REST API call.

I'm sure not all IT departments will allow this workaround, but at least I can confirm that it is working (as long as no other customer in the France region is creating invalid requests, resulting in DoS error messages again).

Please keep us informed about the implementation of the token-throttling so we can switch back to the original region.

Regards,

Thomas

[Anonymous]

Hi all,

we are encountering the same issue on all our environments (DEV/TEST/PROD).

All environments use Data Factory pipeline to call a dataset refresh. Each time a new token based on our service principal is retrieved. For each dataset to refresh we reuse the same pipeline, meaning for each dataset we get a fresh token. The refresh pipeline is run twice in parallel to refresh two different datasets.

Sometimes only one of both fails, other times both fail.

On DEV and TEST we only refresh once a day using this approach, on PROD 5 times a day (which is below the limit of 8 times a day).

This has been working for months now (started in March), and since August 9th this is failing in 6 out of 10 attempts.

The only message we get is: "The request was blocked by DoSP" and a client IP address that is changing.

Any suggestion on how to solve this error?

[AlexanderTikh]

We have also today many such error response in Data Factory pipelines

[v-shex-msft]

HI @MarcosSpeca,
Any looping operations that existed in your embed application? BTW, did these customer works in the same network route? Any trace or 'web scrapy' scripts host on these devices?

Please share more detailed information to help us clarify these.

How to Get Your Question Answered Quickly 

Regards,

Xiaoxin Sheng

[MarcosSpeca]

Hi

We are looking for loops in our app, but seems not to be the case.

At first I thought that was a VPN problem, however the problem ocourr with differente customers (in differents network).

We opened a ticket at Microsoft and the support are investigating the problem. Thanks for replying, though.

[v-shex-msft]

Hi @MarcosSpeca,

You can also take a look at power bi audit logs if any unusual operations recorded.

Track user activities in Power BI - Power BI | Microsoft Docs
Regards,

Xiaoxin Sheng

[lbendlin]

Your requests were too frequent and the Denial Of Service Prevention/Protection tool has deemed them to be DoS, and has blocked them. Do you have network logs to corroborate that?

[MarcosSpeca]

Hi! Thanks for the response. I am not sure that the requests are too frequent..., Because we do not had any scenario change since the error start.  The number of access per day on our app (that embed the report) are the same, same users, and same locations/IPs that we had before.

We do not have any specific network logs, only this error above that we register in our app log.

[kelvinngai]

Hi @MarcosSpeca 

Did you get any feedback from microsoft or is it solved?  

I got the same error even call the powerBI API from data factory for less than 5 times and the next day, it still give me that error.

Thanks a lot

Kelvin

[MarcosSpeca]

Hi @kelvinngai  yes, actualy I had to open a support ticket.

What happend was that the embedd token was expired, and when the embed report tries to get the info with the expired token and got blocked. 

So when our users try to change the report page with the expired token, every visual make a request to the server, and the server understand this "many requests" with the token expired as a risk, and block the user IP.

We renew the token when its about to expire in our app, and the problem was resolved.

I hope this help solve your issue.

[kelvinngai]

Hi Marcos,

Thank you for your reply. For my case, it shows the error in Azure Data Factory, but not in POSTMAN

It is not because the token expired. I use the same token in POSTMAN and call the dataset refresh many times , it works fine.

Maybe kind of network protection from the ADF side. I will try to use new key every API call in Azure data factory

Thanks 

Kelvin Ngai