Forum Discussion
Intermittent DM_GWPipeline_Gateway_CanceledError
We have had a string of refresh failures due to the DM_GWPipeline_Gateway_CanceledError error over the last week or so. The dataset affected varies. Most are scheduled refreshes, but it has happened on manual refreshes as well. Generally, when manually starting the refresh again, it works fine. The sources are working fine in other datasets.
Any idea what could be causing this?
62 Replies
To keep everyone updated...the Microsoft ticket has been escalated and they acknowledge that other companies are seeing the same issue.
We were told to increase our disk space, to see if that resolved the issue. We have done so twice now, and are still getting the issue. Less volume of failures, but still there.
They gave the same response to us, but did not explain why the error occurs only with datasets connected to the SQL Server.
We have datasets connected to Analysis Services that are working normally.Glad I found this thread to see that I'm not alone, but sad to see that there's no solution in sight still.
We've been experiencing these random and frequent refresh failures for two or three weeks now, and it's a huge issue. Which refreshes fail appears random, but it's a ton each day.
I have updated to the latest gateway (March update), restarted the service and the server it lives on, etc. No improvement.
I really pray Microsoft can fix this soon because it's affecting our highest priority reporting and financials company-wide right now.
Quick update:
Saw one user post that rolling back to the September 2024 version of the gateway fixed things for them, so I decided to test it myself.
I installed the September version as a second, separate gateway instance and started pointing a bunch of models that had been consistently failing to refresh with the Gateway Cancelled error for weeks. After testing about 10 of them or so, none of them have failed yet when connecting through the September 2024 gateway.
So I think it's very likely that the underlying issue lies with the last few months of gateway updates. This would ideally need to be fixed in a near-future update so we can apply it to our main install because you can't roll back to an older version without completely uninstalling the newer one first.
For now I'm keeping some of our most important stuff connected to this September test gateway to minimize failures, but we'd really need for a hotfix update to the newer gateway install long-term.
Same situation here. It started the weekend after the Azure incident on East US 2 (don't know if they're related or not) We keep receiving intermitent refresh failures messages DM_GWPipeline_Gateway_CanceledError.
We are on a shared cap, but our semantic models are small (the only one Big is 345Mb) and our workspaces are also small one (between 10 an 100Mb) the only one big is 2Gb (but the limit is 10Gb). Needles to say we have the latest OPDG installed and also performed an upsize on our VMs hosting them, so no perf issues there.
We are experiencing exactly the same issues as you since around january 11th. I have since updated the On Premise Data Gateway, I have added redundancy by creating a cluster and installing another On Premise Data Gateway on a different server. I have reduced the number of concurrent refreshes and removed unecessary refresh schedules. The issue is still persistent and unpredictable affecting random dashboard refresh. Usually manually refreshing works fine. I created another post on this same issue and others have commented that they have the same issue going on.
We have been working with OPDG for years without problems and all of a sudden these issues are happening. This is an MS issue and needs to be resolved by them.
We continue to receive the DM_GWPipeline_Gateway_CanceledError error. Are any of you still seeing this error? Has anyone had a response back from Microsoft? Anyone have insight into what is causing this refresh error? Thanks!!
We were told last week that the issue has been escalated. No details on the cause. I haven't heard anything since then.
Thanks for keeping us in the loop! Im still having many failures every single day. Some days are worse than others. But out of our 100+ reports, maybe random refresh failures per day. I just keep doing manual refreshes whenever necessary.
We are also facing this issue. Even after updating the gateways, upping the compute resources and rebooting the gateway host machines. The ticket is still open with Microsoft. Not really sure when this issue will be resolved.
Hi again, anyone have some response from microsoft?
only read some post similar the solution was rollback to September 2024 Gateway is the correct solution?searched the incident if was fixed but is removed, don't understand.
Link to the microsoft have the issue removed:
Known issue - Intermittent refresh failure through on-premises data gateway - Microsoft Fabric | Microsoft LearnWe continue to see the issue on a daily basis. Microsoft is still investigating.
FYI, the issue mentioned in the link you included is for a different error. The error we get is the DM_GWPipeline_Gateway_CanceledError error.
Hello everyone,
After several tests with Microsoft, our network team discovered an issue with the communication route to the link. I’m not able to explain it very well since I'm not part of the network team, but after we fixed this route, the errors significantly decreased. It’s still not 100% error-free, but it's much better.
We also shared the gateway logs with Microsoft, and they asked us to whitelist some IPs, as they were showing failures in the log.
They suspect that the issue is indeed related to the gateway version. This is all we've discovered in the past few days.
Thank's for the information victor_mirante , our organization is testing and monitoring update april 2025( the update was applied 30th April), i will comment later, with news.
Hi awright19,
Thank you for your response and confirmation. We sincerely apologize for any inconvenience caused.
For any future queries, kindly contact the Microsoft Fabric community. We will be glad to assist you.
Thank you for your understanding and cooperation.
Hi awright19,
Thank you for your reply and confirmation. We sincerely apologise for any inconvenience caused.
For any future queries, please feel free to contact the Microsoft Fabric community. We will be happy to assist you.
Thank you for your understanding and cooperation.
I have also started experiencing this issue intermittently out of the blue. Here is my error message:
{"error":{"code":"DM_GWPipeline_Gateway_CanceledError","pbi.error":{"code":"DM_GWPipeline_Gateway_CanceledError","parameters":{},"details":[],"exceptionCulprit":1}}} Table: Collection Status.
FYI - my issue appears to have been resolved now without any intervention from me. Apparently it was a known issue that Microsoft resolved. Refer to: https://community.fabric.microsoft.com/t5/Service/Data-Refresh-error-quot-DM-GWPipeline-Gateway-CanceledError-quot/m-p/4381851#M262947
Here are the error messages from the 2 that failed overnight last night. This is what I see every time...different tables obviously in different datasets, but otherwise, the same error. For those datasets where it's happened multiple times, it's different tables each time. There's no consistency.
{"error":{"code":"DM_GWPipeline_Gateway_CanceledError","pbi.error":{"code":"DM_GWPipeline_Gateway_CanceledError","parameters":{},"details":[],"exceptionCulprit":1}}} Table: Employees.
{"error":{"code":"DM_GWPipeline_Gateway_CanceledError","pbi.error":{"code":"DM_GWPipeline_Gateway_CanceledError","parameters":{},"details":[],"exceptionCulprit":1}}} Table: NonPublishing Days.
Hi awright19,
Thank you for reaching out to the Microsoft Fabric Community Forum.
Please follow these steps to resolve the issue:
- Check the gateway performance metrics in the Power BI service. Make sure it is updated to the latest version. If you notice high usage, consider increasing the resources allocated to it.
- Adjust dataset refresh timings to avoid multiple datasets refreshing at the same time. This will help reduce the load on the gateway.
- Simplify the queries in the datasets. Breaking large queries into smaller parts can help reduce the time required for refresh.
- Use a separate gateway for important datasets to ensure enough resources are available during refresh.
- Check the network connection between the Power BI service and your data sources to ensure it is stable and does not cause interruptions.
If you find this response helpful, please mark it as the accepted solution and give kudos. This will help others in the community facing similar issues.
Thank you.
We are also experiencing the same error message, intermittently, we did upgrade to the Gateway version 3000.254.3 just before these failures began. Its not the load, we have very few reports and have a load balance. Microsoft, @v-pnaroju-msft I believe the fix is more than what is suggested. Thank you for any help here.
We updated to the latest version of the gateway after the issues started and they have continued after the update. My team added some disk space (no change, in fact we ended up with more failures the next day). These datasets have been refreshing successfully for years on the schedules. While some are happening when there are multiple datasets refreshing, it also happens when there is only 1. The gateway is working because other datasets are refreshing succcessfully.
The frequency of the failures has increased. I came in this morning to 5 failures.
