Forum Discussion
Mirrored Snowflake Database Storage
Hi Folks,
Some info upfront:
- We use a Fabric F16 Capacity
- I know the F16 Capacity offers 16TB free mirroring Storage
I was trying the Mirrored Snowflake Database. It worked well. It was just running for some weeks when I noticed from the "Fabric Capacity Metrics" in the "Storage" Page, it showed in the column "Current Storage (GB)" that the Storage had grown to 18 TB.
This is way bigger than the all the mirrored tables in Snowflake are.
So I started investigating slowly and created a new Mirrored Database. This time I only mirrored one single table.
It started with 2.47 GB, which sounds like a reasonable size for this table. The "Billable Storage (GB)" was stating 0.08 (still not sure what that means since F16 offers 16TB of free mirroring storage).
I was observing for a few day. The size was growing with exactly 2.47 GB each day.
So, I talked to our snowflake data engineers and aparently the rebuild the whole table every night instead of updating it.
With this information I assume the CDC of the Mirrored Snowflake Database thinks everything is new.
Furthermore I assume the Mirrored Snowflake Database holds the "old" records as a backup.
So far so good.
But in the settings of the Mirrored Snowflake Database in "Delta Table Management" the retention threshold is set to 1 day.
Should Fabric not automatically vacuum everything old every day?
And if not, can I trigger the cleanup manually?
Hey Lanceometer ,
You have diagnosed it right, this is a source-side pattern, not a Fabric bug.
- Why storage grows: Fabric mirroring uses Snowflake Streams (CDC). A nightly full rebuild makes every row look new, so a fresh set of Parquet files gets written each cycle while older Delta versions stick around
- 1 day retention: It's only a threshold for when old files become eligible for cleanup, not guarantee that vacuum runs daily
- Manual VACUUM: Not supported on a mirrored database. Maintenance is handled internally (OPTIMIZE runs roughly every ~50 writes per table)
Next steps:
- Real fix: ask the Snowflake team to switch to incremental MERGE/UPDATE instead of a nightly rebuild
- Otherwise, skip mirroring for this table — use a OneLake shortcut or a pipeline load
- Keep retention at 1 day and give it a couple of cycles to plateau
Useful links:
Mirroring Snowflake in Fabric
VACUUM Delta tablesFound this useful? A quick Kudos goes a long way.
Got what you needed? Marking this as the Accepted Solution helps others land on the right answer faster when they search for the same thing
6 Replies
Hey Lanceometer ,
You have diagnosed it right, this is a source-side pattern, not a Fabric bug.
- Why storage grows: Fabric mirroring uses Snowflake Streams (CDC). A nightly full rebuild makes every row look new, so a fresh set of Parquet files gets written each cycle while older Delta versions stick around
- 1 day retention: It's only a threshold for when old files become eligible for cleanup, not guarantee that vacuum runs daily
- Manual VACUUM: Not supported on a mirrored database. Maintenance is handled internally (OPTIMIZE runs roughly every ~50 writes per table)
Next steps:
- Real fix: ask the Snowflake team to switch to incremental MERGE/UPDATE instead of a nightly rebuild
- Otherwise, skip mirroring for this table — use a OneLake shortcut or a pipeline load
- Keep retention at 1 day and give it a couple of cycles to plateau
Useful links:
Mirroring Snowflake in Fabric
VACUUM Delta tablesFound this useful? A quick Kudos goes a long way.
Got what you needed? Marking this as the Accepted Solution helps others land on the right answer faster when they search for the same thingHi Lanceometer ,
Your suspicions are correct: if the table in Snowflake is fully rebuilt every night, the mirrored database CDC interprets everything as new, and the old files stay in storage until VACUUM runs based on the retention threshold.
That said, for this specific pattern (daily full reload at the source), Mirroring might not be the best fit, since it's designed for true incremental changes. Maybe it's worth considering two alternatives that align much better with your scenario:
- Copy Job with CDC from Snowflake, which relies on Snowflake's native change tracking: https://learn.microsoft.com/en-us/fabric/data-factory/cdc-copy-job-snowflake
- A classic Copy activity with the Snowflake connector, where you control the load logic yourself (truncate and load, overwrite, etc.) without depending on the mirrored database's internal retention mechanism: https://learn.microsoft.com/en-us/fabric/data-factory/connector-snowflake-copy-activity
Keep in mind that for incremental approaches (either CDC or watermark-based Copy activity) you'll typically need a reliable column to detect changes, such as a last modified date or an incremental key. Without it, you'd be limited to full loads.
If this helped, please consider giving it a Like. If it solved your issue, please mark it as the Accepted Solution to help others facing the same problem.
Thanks
Hello Lanceometer
Yes, your asumption is correct. Fabric keeps old recods as backup. to answer your question, you can vacumm the old files daily but it does not automatically delete the files. So, Delat tables required explicit vacumme. you can clean up them manullay. Pls read this thread https://learn.microsoft.com/en-us/fabric/data-engineering/delta-lake-vacuum?tabs=sparksql it might helps you.
Thank you!!
Did I answer your question? Mark my post as a solution!
Proud to be a Super User!Hello Lanceometer,
From your investigation, your assumption sounds reasonable.
If the source process rebuilds the entire Snowflake table each night (rather than performing incremental updates), Fabric Mirroring may interpret this as a full set of changes, resulting in new Delta files being created and increased storage consumption.
Regarding the Delta Table Management settings:The retention threshold controls when obsolete Delta files become eligible for cleanup, but it doesn't necessarily mean they are removed immediately after the threshold is reached.
Cleanup (VACUUM) is managed by the Fabric service, and as far as I'm aware, there isn't currently a supported way to manually trigger a VACUUM operation or force cleanup for mirrored databases.
If the storage continues to grow well beyond the expected retention period, despite the retention threshold being set to one day, it may indicate that:
Background cleanup hasn't completed yet, or
The continuous full-table rebuilds are generating Delta files faster than the retention process can reclaim them.
It would also be worth confirming whether the Snowflake ingestion pattern can be changed to perform incremental updates instead of full table rebuilds, as this would significantly reduce both storage growth and processing overhead.
If the storage keeps increasing without stabilizing after several retention cycles, I'd recommend opening a Microsoft support ticket. This would help determine whether the observed growth is expected behavior for mirrored Snowflake databases or if there's an issue with the Delta cleanup process.
Best regards,
Omkar Shinde
Microsoft Fabric Enthusiast | Power BI Consultant
💡 If you found this response helpful, please consider giving it a Kudos.
✅ If this resolves your question, please mark it as the Accepted Solution to help others in the community.Hi Lanceometer,
Thank you for reaching out to Microsoft Fabric Community.
Thank you suparnababu8, Dev_Dholakia, arabalca, Omkar_1712 for the prompt response.
As we haven’t heard back from you, we wanted to kindly follow up to check if the solution provided by the user's for the issue worked? or let us know if you need any further assistance.
Thanks and regards,
Anjan Kumar Chippa
Thank you all,
yes apparently the problem is how the tables are updated in Snowflake via dbt.
I will sit together with our data engineers.
Until the the mirrored database is not an option for us due to our internal dbt build logic or until it is possible to vacuum mirrored dbs in fabric
