Issues with Automated Refreshes on PBI Server - But Works Fine In The App

Hello mjkusler 

Power BI’s integration with Databricks requires the Databricks cluster to be active and accessible during scheduled refreshes. If the cluster is offline or terminated, Power BI cannot retrieve data, leading to refresh failures. This dependency is explicitly noted in multiple sources, which recommend ensuring the cluster is online before scheduled refresh times

Instead of relying on an interactive cluster, you can use a Databricks SQL Warehouse, which allows for querying data without requiring a running cluster

Direct query mode could be an option too

https://telefonicatech.uk/blog/power-bi-with-azure-databricks-for-dummies-in-15-minutes

https://www.element61.be/en/resource/connecting-power-bi-databricks-lakehouse

Please accept the answer if this is useful 

Hi mjkusler 

Your issue likely stems from a timing conflict between Databricks' table update process and Power BI's scheduled refresh. The failure occurs at the same time daily, suggesting that Power BI is querying the table while it's being updated or temporarily unavailable. The error message indicates missing columns, but the specific column varies, implying an inconsistent schema during refresh. Since manual refresh works fine, the table is likely restored after the scheduled refresh fails. Other tables follow the same pipeline but don’t fail, meaning this specific table might have a slower or more complex update process. The S3 to Databricks transfer should be checked for delays, locks, or incomplete data loads at the failure time. Databricks’ metadata caching could also cause Power BI to see an outdated schema. Solutions include adjusting the refresh schedule, ensuring table availability before refresh, or using a staging table to maintain consistency. Adding logging in Databricks can help track when the table is fully updated before Power BI queries it.

Hi mjkusler,

Thanks for reaching out to the Microsoft fabric community forum.

It looks like you are facing issues with failing refreshes on PBI Server. If the solutions provided by Poojara_D12 and nilendraFabric resolves your issue, kindly accept it as solution.

I would also take a moment to thank Poojara_D12 and nilendraFabric, for actively participating in the community forum and for the solutions you’ve been sharing in the community forum. Your contributions make a real difference.

If I misunderstand your needs or you still have problems on it, please feel free to let us know.  

Best Regards,
Hammad.
Community Support Team

If this post helps then please mark it as a solution, so that other members find it more quickly.

Thank you.

Hi mjkusler,

As we haven’t heard back from you, so just following up to our previous message. I'd like to confirm if you've successfully resolved this issue or if you need further help.

If yes, you are welcome to share your workaround and mark it as a solution so that other users can benefit as well. If you find a reply particularly helpful to you, you can also mark it as a solution.

If you still have any questions or need more support, please feel free to let us know. We are more than happy to continue to help you.

Thank you for your patience and look forward to hearing from you.

Anonymous 

We did figure out the issue, but it was actually a coding issue in Python, affecting how data was being loaded into Databricks. We have since solved the issue. So ultimately, this was never a PowerBI issue. It may be best for me to erase this entire thread, if that can be done?

Althought I do appreciate everyones thoughts as it gave me some things to consider in the future.