Forum Discussion

MarcosSpeca's avatar
MarcosSpeca
Frequent Visitor
5 years ago

Embedded The request was blocked by DoSP

Hi, I'm getting the error below when embedding a report: "{"message":"Fail to load - Could not retrieve models and explorations.","detailedMessage":"The request was blocked by DoSP <ClientEndpoint...
Need Help
FGA's avatar
FGA
New Member
5 years ago

 Hi all,

As of 4 days ago we start to receive the same error ("The request was blocked by DoSP ") from our Synapse PBI dataset refresh pipeline. At first it was just once maybe every 4 to 6 hours and now its 8 out of 10 attempts that get denied. Our pipeline ensures that with every attempt we get a new token based on our Service principal with the proper permission and it had been working well for more then a month.  The most frequent running pipeline is every 30 min. and the rest every 6 hours. The frequency is a requirement from our business users. Is there any UI or Microsoft department were we can indicate the PBI Service that our calls are trusted? We need to keep the current frequency and solve this issue. Any suggestion would be very much appreciated.

Regards, Fabian

Anonymous's avatar
Anonymous
Not applicable
5 years ago

We have opened ticket with MS and answer was:

Suggestion
===========
Regarding your concern about this issue, we checked this with our power bi backend team, they indicated that the Power BI algorithm is designed due to security consideration of Denial-of-Service (DoS).
The policy is if the IP address sends more than a certain number of invalid requests within 1 ~ 5 minutes, the IP is blocked for 5 minutes. IP is only blocked when the requests cannot pass authentication.
If someone sends large number of requests with invalid token, PBI needs to query AAD and metadata store to authenticate the user. This can bring down our system or impact other users. while there are many customer share same IP by using ADF in same region. We fully understand it’s not reasonable that you was blocked by other customer in ADF, but from power bi side, we only receive the request and know which IP it’s sending from and which token it bring, if it’s a invalid token, it’s not possible to know who is sending it, the only thing we know is the IP of the request.

 

We have confirmed with our backend team that currently we cannot change this from power bi side about the IP of ADF due to security consideration of Denial-of-Service (DoS), today we got some update from backend team, they are actively working on improving this behavior like only throttle the invalid token.

 

Before that, we may suggest you to use the short-term workaround by changing ADF runtime to different region so that we can use a different IP to unblock this situation if possible.

  • FGA's avatar
    FGA
    New Member
    5 years ago

    Hi Alexander,

    Thanks for this update.

    Do you also happen to know how long it takes for the security algorithm (policy) before the IP address is cleared from black list ?

     

    @Question2community: As there is currently no way to influence the PBI Security algorithm, we are then turning into looking deeper in our calls.
    How can we validate (logs or similiar) what our Synapse pipelines calls to PBI REST API's are actually doing ?
    Things like retrys, warning, invalid tokens, errors, etc. This can maybe then help us to pin point the probable cause.

     

    To the suggestion of Microsoft, changing ADF runtime to different region, we would need to get clearence from our Security team first. Also, I'm curious as if this will unblock permanetly or just let calls through from other region but when swithcing back to current region the algorithm will again block our calls. Do you happen to know the behavior in this case?

     

    Regards, Fabian

    • Anonymous's avatar
      Anonymous
      Not applicable
      5 years ago

      Hi Fabian, Alexander,

       

      I implemented the suggested workaround on our DEV and TEST environment. It is a little bit more work then it seemed originally as you cannot 'move' an existing data factory to another region. Rather you have to recreate it completely in another region, and then apply all access policies and necessary security settings.

      Luckily we did not had to recreate all our data factories in another region, 'moving' the "refresh dataset" data factory to another region was sufficient. (it 'moved' from West Europe to France Central).

      The orchestration data factory (residing in West Europe) started the 'refresh dataset' data factory (residing in France Central) using REST API call.

      I'm sure not all IT departments will allow this workaround, but at least I can confirm that it is working (as long as no other customer in the France region is creating invalid requests, resulting in DoS error messages again).

      Please keep us informed about the implementation of the token-throttling so we can switch back to the original region.

      Regards,

      Thomas