Databricks Queries NOT Hitting Databricks

[sarah_riecke]

I have a report published in the service that connects to Databricks. 

About a month ago we were getting a timeout error, BUT the queries sent from databricks weren't hitting the serverless sql warehouse. We updated the warehouse from serverless to pro, and this seemed to solve the problem for about 3 weeks.. That was until a couple of days ago (the details below are also what we encountered 3 weeks ago before switching from serverless to pro). 

In the am (UTC), we have adf pipelines that kick off to refresh out data in databricks, and then refresh Power BI. (Edit: the report refreshes as part of the ADF pipeline, after everything else finishes refreshing, and the report utilizes a different sql wrehouse than the notebooks refreshing in databricks, they utilize a cluster). 

But with the latest error, we can see the refresh kick off in Power BI, but it tries for 2 hours before failing. (4 times at about 30 mins each - trying to connect to Databricks). 

If I go look at the monitoring of the sql warehouse for the past 24 hours (in which the power bi failure happened), there is nothing that shows up, so the queries aren't even hitting databricks to spin up the sql warehouse. 

We had about 3 weeks of successful refreshes via API but then suddely the same errors again, and nothing changed on the power bi side. 

But when I try to manually refresh later in the data, the refresh succeeds, and it goes through just fine. Initially we thought that other workspaces within the org were using serverless capacity, and that is why we initially changed from serverless to pro, but now we are seeing the same errors with pro again. 

I have tried looking for other solutions out there, but haven't really found much. I'm not sure why in the morning during the scheduled API refresh it won't even hit databricks from power bi, but later in the day it will with a manual refresh. 

Edited to add:
SQL Warehouse Details

[v-sgandrathi]

Hi @sarah_riecke,

Thank you for your suggestion; it’s an important consideration.

Currently, we haven’t set up any activities in the ADF pipeline to scale down or terminate the SQL Warehouse used by Power BI after the ELT process. The pipeline’s notebooks operate on different clusters, and the SQL Warehouse for reporting is separate from those resources.

Your point about warehouse availability during scaling is valid. Since the issue only happens during the early morning pipeline-triggered refresh and not during manual refreshes later, it’s possible the SQL Warehouse is in a stopped or starting state when Power BI tries to connect via the API.

No queries are logged in the SQL Warehouse monitoring during the failure, which suggests the connection attempt fails before reaching Databricks, possibly because the warehouse isn’t fully ready or responsive.

We will check:

If the SQL Warehouse is stopped before the refresh

Whether auto-stop or auto-start settings affect availability

Adding a pre-check step in ADF (such as a lightweight query or explicit start) before the Power BI refresh

Including a short delay after pipeline completion to ensure the warehouse is ready

This should help determine if the issue is related to warehouse readiness during the API-triggered refresh.

Thank you.

[Natarajan_M]

Hi @sarah_riecke , It seems that Power BI is unable to establish a connection to Databricks. Could you please clarify if there is any activity in the ADF pipeline related to scaling down Databricks after the ELT process? The issue may be that the Databricks server is not available for Power BI to connect, which can happen during the scaling up or down of the Databricks server.

[v-sgandrathi]

Hi @sarah_riecke,

Thanks for clarifying. Since the Power BI refresh starts only after the ADF pipeline finishes and the notebooks run on clusters separate from the SQL Warehouse, it seems unlikely there is direct compute contention between the notebook tasks and the warehouse.

Because no queries show up in SQL Warehouse monitoring, it appears the Power BI Service connection isn’t reaching the Databricks SQL endpoint during the failed refresh attempts. The fact that the refresh works later in the day suggests the issue might be due to environment conditions at the time the pipeline triggers the refresh, rather than a configuration change.

It may be helpful to check if multiple pipelines or automated refreshes run around the same time. Even across different workspaces or projects, simultaneous API-triggered refreshes can cause connection delays in Power BI Service. In these cases, the dataset refresh may keep retrying, but the request never reaches Databricks SQL Warehouse, explaining the lack of query history in the logs.

Also, confirm that the SQL Warehouse is fully started when the API refresh request is sent. If it’s still starting up, the connector could retry several times before timing out. Since the refresh later in the day works, it’s possible the warehouse is already active then.

Reviewing the Power BI dataset refresh history and capturing the Activity ID for failed runs may help determine if the issue happens during the initial connection stage. If possible, try triggering the refresh a few minutes after the pipeline completes or run a small validation query against the SQL Warehouse before calling the Power BI refresh API. This can help confirm if the problem is related to warehouse readiness or connection timing.

When automated refresh fails but manual refresh later succeeds, it’s usually due to timing or connection initialization on the service side, rather than an issue with Databricks compute, especially when no activity is recorded in warehouse monitoring.

Thank you.

[v-sgandrathi]

Hi @sarah_riecke,

Based on your description, it seems the scheduled refresh in Power BI is failing before the query reaches Databricks, which is why there’s no activity in the SQL Warehouse monitoring logs. Since the manual refresh works later in the day, the issue probably happens during the connection initialization from Power BI Service, not within Databricks.

This could be due to timing and overlapping workloads in the morning. When your ADF pipelines run to refresh Databricks and Power BI starts its dataset refresh around the same time, Power BI might have trouble connecting to the SQL Warehouse, especially if the warehouse needs to wake up or if other workloads are running. Power BI will retry several times, which matches the multiple retries and eventual failure you’ve noticed.

It’s also worth checking authentication and connection stability. If the Databricks connection uses a personal access token or stored credentials, there might be occasional authentication or connectivity issues during scheduled refresh. Re-validating credentials in the dataset settings can help rule this out.

Since the warehouse shows activity when you run the refresh manually, introducing a short delay between the ADF pipeline completion and the Power BI refresh could help, giving Databricks time to finalize updates and ensuring the warehouse is ready.

Reviewing the dataset refresh history in Power BI Service may also provide more details about the connection failure, such as activity IDs or error messages.

Overall, since the queries don’t appear in the Databricks logs, the issue is likely on the Power BI Service side during the connection stage, rather than with the warehouse configuration.

Thank you.

[sarah_riecke]

The Power BI refresh is part of the ADF pipeline. We do not use the scheduled refresh within Power BI service. It is part of the ADF pipeline, so it refreshes once everything else refreshes, then the pipeline kicks off Power BI refresh off. Power BI isn't starting it's refresh until those are done. So there is no overlap (at least within the same pipeline, I may have to confirm if there is another pipeline that starts to run at the same time). 

But also, assuming there is only 1 pipeline, out report refreshes AFTER the data is refreshed in the pipeline, once that completed then it sends to Power BI to refresh. But also, our notebooks that are refreshing are utilizing different clusters than the SQL Warehouse we use for Power BI.