OneLake storage keeps growing?

Hi Questions_D2i ,

Two things are happening at the same time.

1. What the Capacity Metrics App is showing

The second image you shared displays the 30 days ago accumulation, not the current size of OneLake at a given point in time. This means that even if you reduced your storage to zero today, the line would still be growing — because it reflects the cumulative sum of GB since the 1st of the month. This is the expected behavior of Fabric's billing model, not an issue with your data. (https://learn.microsoft.com/es-es/fabric/enterprise/metrics-app-storage-page#column-charts

)

2. The actual problem: Delta orphan files

With overwrite across 45 tables every night, Delta Lake does not physically delete the previous Parquet files — it marks them as removed in the _delta_log but they remain in OneLake. Those orphan files increase the real daily size, and that larger daily size is what feeds the monthly accumulation.

3. Solution: OPTIMIZE + VACUUM across all three layers every night

OPTIMIZE → compacts the multiple small Parquet files generated by each overwrite into larger files, reducing fragmentation and total size

VACUUM → physically deletes the orphan files that Delta has marked as removed

Always run OPTIMIZE before VACUUM, as OPTIMIZE itself generates new orphan files that VACUUM will clean up in the same pass.

More: 

- Official documentation:
VACUUM in Fabric: https://learn.microsoft.com/fabric/data-engineering/lakehouse-table-maintenance

OPTIMIZE :https://learn.microsoft.com/es-es/fabric/data-engineering/table-compaction?tabs=sparksql#optimize-command 

If you find this helpful please give it a 👍, and if it answered your question please mark it as a solution ✅. This helps the community and motivates me to keep contributing. Thank you!

Hi Questions_D2i,

Hope you're doing well!

Root cause

Even with mode("overwrite"), Delta Lake never deletes old files immediately. It marks them as "deleted" in the transaction log while new Parquet files are written alongside the old ones, until explicitly cleaned up.

Your nightly pipeline stacks one full copy every night. Your 47 GB = 8 nightly copies of your 6 GB actual dataset.

Why VACUUM didn't work ?

1. The safety check blocked it silently Without disabling retentionDurationCheck, VACUUM silently falls back to the 7-day default.

2. OneLake's soft-delete layer retains files for up to 7 days even after Delta removes them logically.

One-time cleanup

from delta.tables import DeltaTable
layers = {
    "LH_Bronze": "abfss://...<Bronze path>...",
    "LH_Silver": "abfss://...<Silver path>...",
    "LH_Gold":   "abfss://...<Gold path>..."
}

spark.conf.set("spark.databricks.delta.retentionDurationCheck.enabled", "False")

for layer, base_path in layers.items():
    for table in mssparkutils.fs.ls(f"{base_path}/Tables"):
        try:
            dt = DeltaTable.forPath(spark, table.path)
            dt.optimize().executeCompaction()
            dt.vacuum(0)
            print(f"✅ {layer} / {table.name}")
        except Exception as e:
            print(f"❌ {e}")

spark.conf.set("spark.databricks.delta.retentionDurationCheck.enabled", "True")

Prevent recurrence by adding to every nightly pipeline

spark.conf.set("spark.databricks.delta.retentionDurationCheck.enabled", "False")
for layer_path in [bronze_path, silver_path, gold_path]:
    for table in mssparkutils.fs.ls(f"{layer_path}/Tables"):
        dt = DeltaTable.forPath(spark, table.path)
        dt.optimize().executeCompaction()
        dt.vacuum(0)
spark.conf.set("spark.databricks.delta.retentionDurationCheck.enabled", "True")

Or permanently via Delta table properties:

spark.sql("""    ALTER TABLE my_table SET TBLPROPERTIES (        'delta.logRetentionDuration' = 'interval 1 days',        'delta.deletedFileRetentionDuration' = 'interval 0 hours'    )""")

Expected results

Billable storage flattens within 24–48h after OneLake's soft-delete window clears. Your architecture is correct. So, this is purely a Delta housekeeping gap.

Hope this helps. Feel free to ask me questions if needed, and don’t forget to give kudos & Accept as Solution if this guidance worked for you. That's motivate me to keep helping.

Best regards,

Oussama (Data Consultant & Fabric's Expert)

Hey Questions_D2i,

You are already running the VACCUUM command with 0 hours retention, which is right.

Also agree with oussamahaimoud on the soft-delete concept. This Microsoft doc explains the topic in bit more detail: https://learn.microsoft.com/en-us/fabric/onelake/onelake-disaster-recovery

]

And this doc covers the billing: https://learn.microsoft.com/en-us/fabric/admin/retention-recovery

To resolve, notebook scripts or API Calls is one of the ways to identify soft-deleted items and remove them. However, my preferred way is to control the retention (extend or permanently disable) directly from the Admin Portal UI:

Additionally, I can view or permanently remove these soft-deleted items through the Recycle Bin UI:

Hope this helps!

Best,

Harshit

Hi Questions_D2i  Let me summarise the answers above (correctly said) in a more succinct way:

1. Overwrite does not mean (immediately) hard delete
   Delta table does NOT delete old files immediately, it marks them as removed (i.e. soft delete), but keeps them physically for history (by default 7 days retention).
   https://www.mssqltips.com/sqlservertip/8201/microsoft-fabric-optimize-delta-tables/
2. VACUUM didn’t work (this is expected)
   Running VACUUM retain 0 hours won't work unless the safety is disabled.

SET spark.databricks.delta.retentionDurationCheck.enabled = false;
VACUUM table_name RETAIN 1 HOURS;​

1. Reduce overwrite pattern (better for long-term)
   Instead of full overwrite, why don't you use incremental loads with CDC / watermark / date partition and only update changed data

In summary, Delta keeps history, overwrite creates new version and storage grows until retention cleanup kicks in. My suggestion is:

• Run forced VACUUM to clean backlog (as an adhoc solution)
• Decide to accept 7-day retention storage OR shorten the retention (safely)
• Move from full overwrite to incremental load

Thankyou, arabalca, oussamahaimoud, stoic-harsh, and rizalard0684 for your responses.

Hi Questions_D2i,

We appreciate your inquiry through the Microsoft Fabric Community Forum.

We would like to inquire whether have you got the chance to check the solutions provided by arabalca, oussamahaimoud, stoic-harsh and rizalard0684 to resolve the issue. We hope the information provided helps to clear the query. Should you have any further queries, kindly feel free to contact the Microsoft Fabric community.

Thank you.

Hi Questions_D2i,

This is a very common issue when managing a Medallion Architecture in Microsoft Fabric! The steady storage growth you are experiencing in your Bronze, Silver, and Gold layers is because Delta tables in Fabric retain all historical parquet files from previous writes to support Delta Time Travel, even when using mode(overwrite).

Since a full overwrite creates a new Delta table version every night while keeping the old ones, the historical data files accumulate in OneLake until they are cleaned up. 

Here is how you can optimize and resolve this across your Medallion layers:

• Bronze (Ingestion layer): Since Bronze gets full night-by-night overwrites from JD Edwards, time travel is rarely needed here. You can safely run VACUUM on Bronze tables with a low retention window (e.g. RETAIN 0 HOURS if zero time-travel is needed) to immediately reclaim storage.
• Silver & Gold (Transformation layers): For Silver and Gold, you might want to retain a few days of history for debugging. Run VACUUM with RETAIN 72 HOURS or RETAIN 168 HOURS (7 days) at the end of your nightly pipeline to keep only the necessary history and automatically purge everything else.
• Nightly Pipeline Cleanup Notebook: Create a simple utility notebook that loops through the tables in your Bronze, Silver, and Gold lakehouses to execute the VACUUM command, and trigger it as the final stage of your orchestrator pipeline.

For a complete architectural guide on best practices for structuring, partitioning, and managing OneLake storage across your Medallion layers, feel free to check out this comprehensive blueprint: dattasable.com/blog/microsoft-fabric-medallion-architecture-guide

Regards,
Datta Sable

Hi Questions_D2i,

We are following up to see if what we shared solved your issue. If you need more support, please reach out to the Microsoft Fabric community.

Thank you.