Forum Discussion

Data-eng-81's avatar
Data-eng-81
New Member
2 months ago
Solved

DM_GWPipeline_Gateway_MashupDataAccessError: "Timeout expired"

I´ve been having the following error during Power BI Service semantic models data refreshes via Azure Devops.

 

Data source error: {"error":{"code":"DM_GWPipeline_Gateway_MashupDataAccessError","pbi.error":{"code":"DM_GWPipeline_Gateway_MashupDataAccessError","parameters":{},"details":[{"code":"DM_ErrorDetailNameCode_UnderlyingErrorCode","detail":{"type":1,"value":"-2147467259"}},{"code":"DM_ErrorDetailNameCode_UnderlyingErrorMessage","detail":{"type":1,"value":"Timeout expired. The operation did not complete in the allotted time."}},{"code":"DM_ErrorDetailNameCode_UnderlyingHResult","detail":{"type":1,"value":"-2147467259"}},{"code":"Microsoft.Data.Mashup.HostingError.Reason","detail":{"type":1,"value":"Timed out"}}],"exceptionCulprit":1}} "


I understang that it's related to M language queries. But i would like to understand the detail ""Timeout expired. The operation did not complete in the allotted time." What does it means?
 Nowadays, just one of the semantic models tables uses M language. The rest of the tables are brought via SQL to the semantic models.
If I use SQL to bring all the tables to build the semantic model, won´t i have this problem anymore?

Thanks in advance for the help.

  • PauReis's avatar
    PauReis
    2 months ago

    Hi Data-eng-81,

    Fair enough on the command timeout, and your one minute figure does argue against it. Two things about the retry plan before you build it, both documented and both easy to walk into.

    The first is that retrying harder can be invisible. Microsoft's own note on the refresh API says Power BI might drop requests if too many arrive in a short period, that it performs one refresh, queues the next request and drops all the others, and that by design you cannot query the status of dropped requests. On top of that the service accepts only one refresh operation at a time for a given model, and a second request while one is already running returns a 400. So if the trigger really is everything firing at once out of DevOps, tighter retry intervals can produce attempts your PowerShell has no way to see, and the run will look like it failed more often than it did.

    The second is that if you want to test the concurrency idea rather than infer it, the gateway is where to look, and that fits your capacity metrics showing no breach at the time of failure. The number of mashup containers is what determines how many queries can execute in parallel, and it is set by MashupDefaultPoolContainerMaxCount, which has scaled automatically from CPU capacity since the June 2019 gateway release. Counting how many Mashup container processes are alive on that host during a failing window will tell you whether you are saturating it.

    One trap if you do ask your vendor to raise it. Changing those values by hand has no effect unless MashupDisableContainerAutoConfig is also set to True in the same config file. Otherwise auto scaling keeps overriding what you set and it looks like the change did nothing, which is an easy afternoon to lose.

9 Replies

  • The error means the mashup engine hit its wait limit before the source returned all the rows, so the refresh was aborted mid-read. The HResult -2147467259 is the generic OLE DB "operation failed" code that surfaces when the source layer cannot deliver results in the allotted time, and it usually points to either a slow source query or M steps that broke query folding and forced Power Query to process rows locally.

    Moving every table to SQL will not automatically solve it. What matters is that the work stays on the database side rather than getting pulled into the mashup engine. Open the M query, right-click the last applied step, and check View Native Query. If it is greyed out on your M-based table, folding broke somewhere, and the fix is to move that logic into a SQL view or the source query so folding is preserved end to end.

    You can also raise the command timeout in the source options, but that only pushes the ceiling higher, it does not remove the underlying reason the refresh is slow.

     

    If it solved your issue, kindly mark it as the accepted solution and give it a kudos.

     

    Best,
    Shai Karmani

     

    Let's connect in LinkedIn

    • Data-eng-81's avatar
      Data-eng-81
      New Member

      Hi Shai_Karmani

      thank you for your contribution too

      but, as i told to PauReis, I don´t think the command timeout is going to help in this particular situation, because the error occurs (more or less) in one minute (as you can see the below image)

      It seems to me that it is a problem about concurrency (because of the time of one minute) as various data refreshs are triggered at the same moment. All the errors presented the same pattern. What do you think? 

      But thank you for your explanation 

  • Hi Data-eng-81,

     

    On your second question, moving everything to SQL won't make this go away, and it's worth knowing why before you spend a week on it.

     

    There's no path that skips the mashup engine. The docs put it plainly: "Power BI semantic models rely on Power Query to access and retrieve source data." When you pick tables through a SQL connector, Power Query still generates M for them, you just never look at it. So a table you brought in "via SQL" and a table where you wrote M by hand go through the same engine and can both throw this. The M language isn't the culprit here, it's just the layer that reports the error.

     

    On what the message means: the DM_GWPipeline_Gateway_ prefix is the gateway telling you it gave up waiting on the source. The -2147467259 is a generic failure code, it doesn't narrow anything down, so don't read much into it.

     

    The number that usually matters is the command timeout, and it's set per connector rather than globally. On the connectors that expose it the advanced Command timeout option defaults to 10 minutes, and you can set it in the query itself:

     

    Source = Sql.Database("yourserver", "yourdb", [CommandTimeout=#duration(0, 2, 0, 0)])

     

    That's two hours. Swap the function for whichever connector you're on. I'd put it on the one slow table rather than everywhere, and treat it as buying time rather than fixing the cause.

     

    Do you know which table it's actually dying on? Refresh history will tell you, and if it turns out to be the single M table then Shai_Karmani's folding point is where I'd go next.

    • Data-eng-81's avatar
      Data-eng-81
      New Member

      Hi PauReis 

      First of all, thank you for your contribution.

      Thinking about of what you have said...

      I don´t think the command timeout is going to help in this particular situation, because the error occurs (more or less) in one minute (as you can see the below image)

      It seems to me that it is a problem about concurrency (because of the time of one minute). All the errors presented the same pattern

      But thank you for your explanation. I didn´t know that there's no path that skips the mashup engine

  • v-kathullac's avatar
    v-kathullac
    Icon for Community Support rankCommunity Support

    Hi Data-eng-81​,

    Based on the fact that the errors are consistently occurring at around the one-minute mark, I agree that the command timeout may not address the underlying issue in this particular scenario. The consistent timing does make concurrency or another service-side limitation worth investigating further.

    As the Mashup Engine is involved in the processing path, it would be helpful to investigate whether there are any concurrency, resource, or capacity-related limitations being reached at that point.

    I would recommend monitoring the refresh/query activity around the time of the failure and checking whether multiple operations are being executed concurrently. If the issue persists with the same one-minute pattern, raising a Microsoft support ticket may be the best next step, as Microsoft would be able to investigate the backend/service-side logs and confirm the exact cause.

    Thanks,

    Chaithanya.

  • Hi v-kathullac, thanks for joining this discussion topic.

    It's nice to know that i am on good path to solve this problem.

    About capacity-related limitations,  the fabric capacity metrics report show that, at the time of failure, we didn't exceed the capacity limits.

    I am going to try to improve the data refresh task on powershell, putting intern retry count on the same API calls, setting intervals between retries and, maybe, doing second API call if there area still some failures on data refreshs.

    Nowadays, the gateway is on a server that is managed by one of our vendors. You said that "Microsoft would be able to investigate the backend/service-side logs and confirm the exact cause."

     Would microsoft need the logs of gateway-server-side too?

    Thanks again. I am learning a lot with all comments

    • PauReis's avatar
      PauReis
      Icon for Resolver III rankResolver III

      Hi Data-eng-81,

      Fair enough on the command timeout, and your one minute figure does argue against it. Two things about the retry plan before you build it, both documented and both easy to walk into.

      The first is that retrying harder can be invisible. Microsoft's own note on the refresh API says Power BI might drop requests if too many arrive in a short period, that it performs one refresh, queues the next request and drops all the others, and that by design you cannot query the status of dropped requests. On top of that the service accepts only one refresh operation at a time for a given model, and a second request while one is already running returns a 400. So if the trigger really is everything firing at once out of DevOps, tighter retry intervals can produce attempts your PowerShell has no way to see, and the run will look like it failed more often than it did.

      The second is that if you want to test the concurrency idea rather than infer it, the gateway is where to look, and that fits your capacity metrics showing no breach at the time of failure. The number of mashup containers is what determines how many queries can execute in parallel, and it is set by MashupDefaultPoolContainerMaxCount, which has scaled automatically from CPU capacity since the June 2019 gateway release. Counting how many Mashup container processes are alive on that host during a failing window will tell you whether you are saturating it.

      One trap if you do ask your vendor to raise it. Changing those values by hand has no effect unless MashupDisableContainerAutoConfig is also set to True in the same config file. Otherwise auto scaling keeps overriding what you set and it looks like the change did nothing, which is an easy afternoon to lose.

  • v-kathullac's avatar
    v-kathullac
    Icon for Community Support rankCommunity Support

    Hi Data-eng-81​,

    As we haven’t heard back from you, we wanted to kindly follow up to check if the solution provided for the issue worked? or Let us know if you need any further assistance?

    Regards,

    Chaithanya