Missing data when replugging model

[Victormar]

Hello community,

I am facing a very odd situation, and I feel I have tried everything I could and it still doesn't work.

I have a report that is connected to Azure Databricks, to one database where I developed it. Two weeks ago we created a replica of this database because we are changing the structure in Databricks; So, I replugged the report to this new database, and I lost some data.

We have one main dimension table, which has the vehicles and it's keys. Then we have fact tabels with measurements like mileage, speed, and so. They connect in the model using keys.

When replugged, the vehicle ids from the different tables don't match. I removed all the filters and conections and still it doesn't work.

Just to be clear, if I get the old report, connected to the old database, and I refresh it, all works fine. If I add the new database via parameter, and try to switch between both of them, then the report seems to crash. 

Could it be a problem because both databases are connected to the same cluster in Databricks? I have no idea to what to try anymore.

Thanks in advance for any help 🙂

[Victormar]

Hello!

Thanks for your reply, really good to know.

It turns out in the end the problem was that I had two functions to change between environments, and I only updated one, I totally missed the other one (was hidden between the tables ;P)

But your explanation @Anonymous it's super good to know, thanks 🙂

[Anonymous · Accepted solution]

Hi @Victormar ，
Based on your description that you changed the structure in Databricks, the first thing we need to make clear is that you need to make sure that the schema of the new database is the same as the old one. Any differences in table structures, data types or relationships can cause problems when switching databases. Secondly, since both databases are connected to the same cluster, there may be caching issues. Try clearing the cache in Databricks to ensure that the latest data is used. Finally, after switching to the new database, perform a full data refresh. This helps to synchronize the data and resolve any discrepancies.
CLEAR CACHE - Azure Databricks - Databricks SQL | Microsoft Learn

Best regards,
Albert He

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly