Forum Discussion
Storage increasing daily
You should use Python notebooks. It is so easy top use bronze --> silver --> gold layer. You should use OPTIMIZE commands for indexing and Vaccum commands as well. That way You can apply VACUUM and you can save money a lot.
Thank you for reaching out to the Microsoft Fabric Forum Community.
BhaveshPatel tayloramy Thanks for the inputs. user inputs are usefull.
Additon to users points, Even with VACUUM 0, the higher storage usage can still be normal in your scenario. Since your pipelines run 12 times a day across Bronze, Silver, and Gold, every MERGE/upsert creates new Delta/Parquet files, and over time this can lead to lots of small active files. VACUUM helps clean up old unused files, but it doesn’t combine or shrink the active ones.
Also, Azure Storage Explorer shows the total physical storage being used, not just the actual current table data. That includes active data files, Delta logs, temporary/staging files, and storage across all three lakehouses. So the 16 GB vs 1–2 GB difference is likely due to a mix of frequent small file creation, metadata overhead, and storage across the full pipeline not just old retained data.
If there are any deviations from your expectation please let us know we are happy to address.
Thanks.
Hi,
Thanks For the respnse,
So what is the solution for is there any way to reduce the small or can we delete this files just like vaccum
Hi shirishmathanka
As per my knowledge, Vacuum only removes obsolete/unreferenced files, it does not compact active files. Frequent MERGE/upserts create many small active files across Bronze, Silver, and Gold, which is why storage shows more than actual data.To reduce storage and improve performance, run OPTIMIZE to compact small files, and consider batching updates or reducing write frequency.
I have included the official Microsoft documentation for your review.
Lakehouse and Delta Tables - Microsoft Fabric | Microsoft Learn
Remove unused data files with vacuum - Azure Databricks | Microsoft LearnIf there are any deviations from your expectation please let us know we are happy to address.
Thanks.
Thank you for reaching out to the Microsoft Fabric Forum Community.
I hope the information provided was helpful. If you still have questions, please don't hesitate to reach out to the community.