Forum Discussion
System data in OneLake storage keeps growing for Mirrored Azure Databricks Catalog
In a workspace on our Fabric Capacity we have mirrored an Azure Databricks Catalog. The system data steadily keeps growing day by day and the information given is sparse and doesn't go into detail whether or not it can be reduced.
The number of tables synced hasn't changed since first setup (30) and altough the current storage consumption is small, the fact that it has grown from a few MB to over a GB in the span of a month is quite worrying. Are there any documentation to whether or not the system data can be reduced or can I expect this to keep growing indefinetly?
For context, the catalog and schema being mirrored is our gold layer in databricks that gets updated daily via batch. The tables are "managed tables" in databricks so optmization and vaccum is performed by the system. Retention period is 7 days for the delta tables.
What you're seeing is generally expected behavior for mirrored Delta tables, as system data includes metadata, synchronization state, transaction logs, indexing, and other internal artifacts that Fabric maintains to support mirroring.
A few points to consider:
If your Databricks Gold layer is updated daily, Fabric will continue tracking new Delta transactions and synchronization metadata, so some growth in system data is expected.
Delta VACUUM in Databricks primarily removes obsolete data files after the retention period, but it doesn't necessarily reduce all of the metadata or internal state maintained by Fabric for the mirrored catalog.
If the growth continues well beyond the expected retention window (for example, it keeps increasing linearly even though the data volume and number of tables remain stable), it may indicate that metadata is not being cleaned up as expected.
I'd recommend monitoring the storage for another retention cycle (7–14 days). If the system data continues to grow without stabilizing, it's worth opening a Microsoft Support ticket so the product team can verify whether this is expected behavior or a potential issue with metadata cleanup.
It would also help to know:
Is the reported growth entirely under System data, or is Data storage increasing as well?
Does the growth continue even on days when no changes are made to the mirrored tables?
Have you noticed any correlation between the daily sync schedule and the increase in system storage?
If Microsoft confirms whether this metadata is automatically reclaimed or provides guidance on managing it, please consider sharing the resolution here—it would be valuable for others using Mirrored Azure Databricks Catalogs.
If you found this reply helpful, please consider giving it a Kudos. If it answers your question, marking it as the Accepted Solution will help other community members with the same concern.
4 Replies
- Prince0011
Memorable Member
What you're seeing is generally expected behavior for mirrored Delta tables, as system data includes metadata, synchronization state, transaction logs, indexing, and other internal artifacts that Fabric maintains to support mirroring.
A few points to consider:
If your Databricks Gold layer is updated daily, Fabric will continue tracking new Delta transactions and synchronization metadata, so some growth in system data is expected.
Delta VACUUM in Databricks primarily removes obsolete data files after the retention period, but it doesn't necessarily reduce all of the metadata or internal state maintained by Fabric for the mirrored catalog.
If the growth continues well beyond the expected retention window (for example, it keeps increasing linearly even though the data volume and number of tables remain stable), it may indicate that metadata is not being cleaned up as expected.
I'd recommend monitoring the storage for another retention cycle (7–14 days). If the system data continues to grow without stabilizing, it's worth opening a Microsoft Support ticket so the product team can verify whether this is expected behavior or a potential issue with metadata cleanup.
It would also help to know:
Is the reported growth entirely under System data, or is Data storage increasing as well?
Does the growth continue even on days when no changes are made to the mirrored tables?
Have you noticed any correlation between the daily sync schedule and the increase in system storage?
If Microsoft confirms whether this metadata is automatically reclaimed or provides guidance on managing it, please consider sharing the resolution here—it would be valuable for others using Mirrored Azure Databricks Catalogs.
If you found this reply helpful, please consider giving it a Kudos. If it answers your question, marking it as the Accepted Solution will help other community members with the same concern.
An update to the situation is that the system data all of a sudden dropped to 100 MB from 1.38 GB. My experience is that it in this case didn't follow the retention cycle of 7 days for the delta tables. Perhaps an indication of issues with metadata cleanup as you mentioned. I will continue to monitor the situation but hopefully this was just a case of stale metadata.
Thanks for the reply tayloramy.
The system data is used for the mirroring logs, which will grow at a constant rate as long as your tables don't change.
By change I take it as you mean unsyncing a few tables will reduce the rate of growth at least. However, there is no direct method to reduce the system data while keeping existing number of tables? An equivalent to VACUUM command but for the system data?
