Issues with Automated Refreshes on PBI Server - But Works Fine In The App

[mjkusler]

I have several reports that have up to 30 different data sources. However, several of my reports have the same issue, all with the same data table, which resides in Databricks like all others I connect to.

The issue is everyday at the same time my automated refreshes on the PBI Server fail and it's always the same table and I recieve this message. 

Failure details: The last refresh attempt failed because of an internal service error. This is usually a transient issue. If you try again later and still see this message, contact support. The '<oii>XXXXX</oii>' column does not exist in the rowset. Table: Export_ECG

*The red text is because it's always a different column, but the point is that it simply sees an empty table.

The interesting part of all this is, if I open the PBIX file in the app and do a manual refresh it works just fine. Then I republish and the rest of the refreshes work all day until the next morning when it breaks and the cycle starts over again.

We have examined the Python code that is used to move the tables from an S3 bucket to Databricks and have found no issues. FYI every table I use follows this same process. However, we can not figure out why this table is the only one that has this issue.

Has anyone encountered anything like this before?

[mjkusler]

@v-mdharahman 

We did figure out the issue, but it was actually a coding issue in Python, affecting how data was being loaded into Databricks. We have since solved the issue. So ultimately, this was never a PowerBI issue. It may be best for me to erase this entire thread, if that can be done?

Althought I do appreciate everyones thoughts as it gave me some things to consider in the future.

[v-mdharahman]

Hi @mjkusler,

I'm glad that your issue has been resolved. Can you please mark the helpful reply and accept it as the solution. This will be helpful for other community members who have similar problems to solve it faster.

Thank you.

[v-mdharahman]

Hi @mjkusler,

As we haven’t heard back from you, so just following up to our previous message. I'd like to confirm if you've successfully resolved this issue or if you need further help.

If yes, you are welcome to share your workaround and mark it as a solution so that other users can benefit as well. If you find a reply particularly helpful to you, you can also mark it as a solution.

If you still have any questions or need more support, please feel free to let us know. We are more than happy to continue to help you.

Thank you for your patience and look forward to hearing from you.

[v-mdharahman]

Hi @mjkusler,

Thanks for reaching out to the Microsoft fabric community forum.

It looks like you are facing issues with failing refreshes on PBI Server. If the solutions provided by @Poojara_D12 and @nilendraFabric resolves your issue, kindly accept it as solution.

I would also take a moment to thank @Poojara_D12 and @nilendraFabric, for actively participating in the community forum and for the solutions you’ve been sharing in the community forum. Your contributions make a real difference.

If I misunderstand your needs or you still have problems on it, please feel free to let us know.  

Best Regards,
Hammad.
Community Support Team

If this post helps then please mark it as a solution, so that other members find it more quickly.

Thank you.

[Poojara_D12]

Hi @mjkusler 

Your issue likely stems from a timing conflict between Databricks' table update process and Power BI's scheduled refresh. The failure occurs at the same time daily, suggesting that Power BI is querying the table while it's being updated or temporarily unavailable. The error message indicates missing columns, but the specific column varies, implying an inconsistent schema during refresh. Since manual refresh works fine, the table is likely restored after the scheduled refresh fails. Other tables follow the same pipeline but don’t fail, meaning this specific table might have a slower or more complex update process. The S3 to Databricks transfer should be checked for delays, locks, or incomplete data loads at the failure time. Databricks’ metadata caching could also cause Power BI to see an outdated schema. Solutions include adjusting the refresh schedule, ensuring table availability before refresh, or using a staging table to maintain consistency. Adding logging in Databricks can help track when the table is fully updated before Power BI queries it.

 

[mjkusler]

@Poojara_D12 @ I appreicate the response. However, this has all been explored and tested including various different settings, configuartions, code changes, refreshing schedules changes etc. Still no luck

[nilendraFabric]

Hello @mjkusler 

Power BI’s integration with Databricks requires the Databricks cluster to be active and accessible during scheduled refreshes. If the cluster is offline or terminated, Power BI cannot retrieve data, leading to refresh failures. This dependency is explicitly noted in multiple sources, which recommend ensuring the cluster is online before scheduled refresh times

Instead of relying on an interactive cluster, you can use a Databricks SQL Warehouse, which allows for querying data without requiring a running cluster

Direct query mode could be an option too

https://telefonicatech.uk/blog/power-bi-with-azure-databricks-for-dummies-in-15-minutes

https://www.element61.be/en/resource/connecting-power-bi-databricks-lakehouse

Please accept the answer if this is useful 

[mjkusler]

@nilendraFabric I appreicate the response. However, this has all been explored and tested including various different settings, configuartions, code changes, refreshing schedules changes etc. Still no luck.

[v-mdharahman · Accepted solution]

Hi @mjkusler,

Even after trying various approaches, If you are still unable to solve the issue you can reach out to Support by raising a ticket with Microsoft Support.

Please refer below link on how to raise a contact support or support ticket.

How to create a Fabric and Power BI Support ticket - Power BI | Microsoft Learn

If I misunderstand your needs or you still have problems on it, please feel free to let us know.  

Best Regards,
Hammad.
Community Support Team

If this post helps then please mark it as a solution, so that other members find it more quickly.

Thank you.