Forum Discussion
Power BI Service refresh taking 4–5 hours, while Desktop takes 5–10 mins
Has anyone else faced this?
I have around 14–16 Import tables in my reports (2–3 PostgreSQL, rest ClickHouse), with ~100 tables total including small lookup/parameter tables. PPU + on-prem gateway on a strong machine. Gateway is updated, and the ClickHouse server is also powerful with good resources.
Desktop refresh is consistently 5–10 mins, but Power BI Service takes 4–5 hours. One recent refresh took 4h 19m, another 4h 22m, and one hit the 5-hour timeout.
What’s confusing is that similar reports using the same setup used to refresh in under 30 mins a few months ago.
I've already tried cloud vs on-prem gateway and Incremental Refresh without much improvement.
Could this be PPU/concurrency/refresh queuing, or is there something specific with ClickHouse + Power BI Service that I should look into?
What would you check first to figure out where those 4+ hours are actually being spent?
VIKASRP You asked what you would check first, so I will answer that literally rather than add a fifth theory to the pile, because you have already spent real effort on two that did not pay.
First check, one minute, and Olufemi7 has you starting in the right place. What is worth adding is what to read once you are in Refresh history, because the total is not the useful number. Look at how a single attempt is broken down. Every refresh is split into two operations that run in sequence: Data, which loads data into the model, and Query Cache, which is the Premium query cache and dashboard tile refresh. They are reported separately. If your 4h19m is nearly all Data, the time is at the source or in transit and tayloramy has you pointed the right way. If a real slice of it is Query Cache, you are looking at something else entirely, and something that cannot happen in Desktop at all.
That second case deserves a serious look given your own history. The documentation is blunt about it: "Significant use of dashboard tiles or premium caching can increase refresh duration, since either can queue many queries after each refresh." Dashboards accumulate quietly. "It used to run in under 30 minutes a few months ago" and "a lot more tiles have been pinned since then" fit together without anything having gone wrong on the ClickHouse side, and it would explain why swapping between a cloud connection and the on-premises gateway changed nothing.
Second check, and this is the one that actually answers where the hours went. On the refresh detail page there is an Execution details column with a Show link, which gives you execution metrics for each refresh attempt. That data used to need Log Analytics or workspace monitoring to get at and it is now in the product. You can reach it from the semantic model refresh settings, from the semantic model details page, or from the Monitoring hub. That turns "four hours" into a shape you can read instead of a number you have to guess about.
Third, and only after those two. You are on PPU, so you have the XMLA endpoint, and it gives you two useful things. You can refresh one table at a time from SSMS or Tabular Editor and time each of them, which converts "16 tables take four hours" into a sorted list. And on the timeout specifically, the five hour cap is a service limit rather than a limit on your model: "On Premium, the maximum refresh duration is five hours, but using XMLA endpoint to refresh data can bypass the five-hour limit." That is not a fix, but it stops the run dying at hour five while you are still trying to measure it.
One thing I would rule out early, because your table count makes it likely. You have around 100 tables and only 14 to 16 that load. If a lot of those lookup and parameter queries reference a shared query rather than going to the source themselves, that shared query is not evaluated once. The docs are unusually direct that this is a common wrong assumption: "it's often assumed that Power Query retrieves the Query1 result, and that it's reused by referenced queries. This thinking is incorrect." Each referencing query re-runs the whole chain beneath it. The same page notes Power Query can cache but may still fetch from the source more than once, which is why an interactive Desktop session that has already pulled everything is not a fair comparison for a cold service refresh. Query Dependencies view will tell you in thirty seconds whether you have that shape, and if you do, the documented fix on PPU is to move the shared query into a dataflow.
Which of the two, Data or Query Cache, is eating the four hours?
Hi VIKASRP
I have seen this happen in the past and what it could potentially be is purely a bandwidth issue from your gateway server into the Power BI service and that is why it is taking a significantly longer time to process. Another option to check is to use the link below from Phil Seamark to understand when the refresh is happening, which component is taking the longest amount of time to help you troubleshoot where to go and look to resolve the long issue. I have customers who have been importing millions of rows that can take a few minutes to load, so it is definitely possible to have a fast refresh. https://dax.tips/2023/12/05/visualize-power-bi-refresh-using-sempy/ https://dax.tips/2021/05/11/visualise-your-power-bi-refresh-in-real-time/
6 Replies
- tayloramy
Super User
Hi VIKASRP,
The first thing I would check is the gateway server, does it have sufficient resources? You can check the logs to see where the query is being hung up. I've seen this sort of behaviour before when the gateway gets put under more load, or when another service running on the same server takes up a lot of RAM. The gateway will then page to disk rather than to RAM, which is very slow.
Another thing to check is how long the query runs on the source. If possible, I would test this from the gateway server, as that is where the query gets issued from when the refresh happens in the service. Potentially there was a network change which makes it slower on the gateway server but not on your system.
You can also check the refresh history in the service, which will tell you how much time was spent on the data load vs on building a query cache, but the gateway logs are going to have the most information (and they happen to be the hardest to read 😞 )
- PauReis
Resolver II
VIKASRP You asked what you would check first, so I will answer that literally rather than add a fifth theory to the pile, because you have already spent real effort on two that did not pay.
First check, one minute, and Olufemi7 has you starting in the right place. What is worth adding is what to read once you are in Refresh history, because the total is not the useful number. Look at how a single attempt is broken down. Every refresh is split into two operations that run in sequence: Data, which loads data into the model, and Query Cache, which is the Premium query cache and dashboard tile refresh. They are reported separately. If your 4h19m is nearly all Data, the time is at the source or in transit and tayloramy has you pointed the right way. If a real slice of it is Query Cache, you are looking at something else entirely, and something that cannot happen in Desktop at all.
That second case deserves a serious look given your own history. The documentation is blunt about it: "Significant use of dashboard tiles or premium caching can increase refresh duration, since either can queue many queries after each refresh." Dashboards accumulate quietly. "It used to run in under 30 minutes a few months ago" and "a lot more tiles have been pinned since then" fit together without anything having gone wrong on the ClickHouse side, and it would explain why swapping between a cloud connection and the on-premises gateway changed nothing.
Second check, and this is the one that actually answers where the hours went. On the refresh detail page there is an Execution details column with a Show link, which gives you execution metrics for each refresh attempt. That data used to need Log Analytics or workspace monitoring to get at and it is now in the product. You can reach it from the semantic model refresh settings, from the semantic model details page, or from the Monitoring hub. That turns "four hours" into a shape you can read instead of a number you have to guess about.
Third, and only after those two. You are on PPU, so you have the XMLA endpoint, and it gives you two useful things. You can refresh one table at a time from SSMS or Tabular Editor and time each of them, which converts "16 tables take four hours" into a sorted list. And on the timeout specifically, the five hour cap is a service limit rather than a limit on your model: "On Premium, the maximum refresh duration is five hours, but using XMLA endpoint to refresh data can bypass the five-hour limit." That is not a fix, but it stops the run dying at hour five while you are still trying to measure it.
One thing I would rule out early, because your table count makes it likely. You have around 100 tables and only 14 to 16 that load. If a lot of those lookup and parameter queries reference a shared query rather than going to the source themselves, that shared query is not evaluated once. The docs are unusually direct that this is a common wrong assumption: "it's often assumed that Power Query retrieves the Query1 result, and that it's reused by referenced queries. This thinking is incorrect." Each referencing query re-runs the whole chain beneath it. The same page notes Power Query can cache but may still fetch from the source more than once, which is why an interactive Desktop session that has already pulled everything is not a fair comparison for a cold service refresh. Query Dependencies view will tell you in thirty seconds whether you have that shape, and if you do, the documented fix on PPU is to move the shared query into a dataflow.
Which of the two, Data or Query Cache, is eating the four hours?
- VIKASRPNew Member
Hi, Thanks for Getting Back On this, The most Hours are taken in the Data Part, I have a good powerful Machine for the Gateway. And Bandwidth is Good as well! Not the Best But its Good
eg,,
Details
Type Start End Duration Status
Data 7/8/2026, 3:09:05 pm 7/8/2026, 7:31:35 pm 4h 22m 29s Completed
Query Cache 7/8/2026, 7:31:35 pm 7/8/2026, 7:31:35 pm Less than 1s Completed- PauReis
Resolver II
Hi VIKASRP,
That split settles it. Query Cache came back under a second, so caching isn't the problem and you can ignore that half of what I said. All 4h 22m is in Data.
With a strong machine and decent bandwidth, the next suspect is what the gateway does before it sends anything back. By default it writes the whole result to disk first: "the on-premises data gateway spools data before returning it to the semantic model, potentially causing slower performance during data load and refresh operations." You can override that. In Microsoft.PowerBI.DataMovement.Pipeline.GatewayCore.dll.config set StreamBeforeRequestCompletes to True, then restart the gateway from Service Settings.
One caveat, since you said the bandwidth is good but not the best. Microsoft warns that streaming can cause reliability issues on slow sources or unstable connections, so try it on this one model and watch for failures before rolling it out.
Before you change anything though, the gateway has already logged where the time went. Performance logging is on by default, and the Query Execution Report splits each query into SpoolingDiskWritingDuration and DataReadingAndSerializationDuration. That tells you whether the four hours is the database handing over rows or the gateway writing them to disk. Those are different problems with different fixes.
Which of the two is bigger?
- GilbertQ
Super User
Hi VIKASRP
I have seen this happen in the past and what it could potentially be is purely a bandwidth issue from your gateway server into the Power BI service and that is why it is taking a significantly longer time to process. Another option to check is to use the link below from Phil Seamark to understand when the refresh is happening, which component is taking the longest amount of time to help you troubleshoot where to go and look to resolve the long issue. I have customers who have been importing millions of rows that can take a few minutes to load, so it is definitely possible to have a fast refresh. https://dax.tips/2023/12/05/visualize-power-bi-refresh-using-sempy/ https://dax.tips/2021/05/11/visualise-your-power-bi-refresh-in-real-time/
- Olufemi7
Super User
Hello VIKASRP,
I will start with the refresh history and then check the gateway.
The Service refresh can involve different refresh activities, and Microsoft notes that PPU refreshes are subject to capacity resources and can queue when resources are constrained.
Microsoft Learn: Data refresh in Power BISince this model uses an on-premises gateway, I will also check the gateway performance during the refresh and compare the source query timings from the gateway machine.
Microsoft Learn: Monitor and optimize on-premises data gateway performanceWith Desktop completing in 5–10 minutes, those checks should help narrow down whether the delay is on the Service/capacity side, gateway side, or source side.