Forum Discussion

JVDA98's avatar
JVDA98
Frequent Visitor
1 day ago

Onelake Storage Report

Hi All,

I checked the OneLake storage Report for the first time, and it blew my mind...
I have one warehouse that is > 1+ TB, while it only contains two (!!) tables.
- 700K rows
- 1.2M rows

I dived deeper into it by connecting the warehouse to blob storage, and I noticed in the subfolder
onelake/xxxxx/xxxxxx/Files/ ; there is OVER 800 GB OF FILES !

The stored procedures of those two are quite complex with different update statements, but this should not generate so much files ; as we are paying them as well.

I already changed the time_travel_retention_cutoff_date, from 30 --> 5 days, but this has no impact on the /files, only the /tables from what I've read (after 36h still no impact there as well though)

The files are kept into that folder going two months back;

- Is there a way to change this setting?
- Is there a way to reduce all those files that are written?
- Anyone else noticing this?

 

data warehouse
Warehouse

2 Replies

  • v-achippa's avatar
    v-achippa
    Community Support
    1 day ago

    Hi JVDA98​,

    Thank you for reaching out to Microsoft Fabric Community.

    The high storage is most likely caused by the multiple UPDATE operations in the warehouse. The storage consumed is not based only on the current number of rows.

    • Changing the retention period from 30 days to 5 days is the correct approach to reduce historical storage.
    • The expired data is cleaned up asynchronously in the background, so the storage will not decrease immediately. There is no need to manually delete the files under OneLake they are managed by fabric warehouse.
    • Please allow the background cleanup to complete and then refresh the OneLake storage report. Also review the stored procedures and reduce unnecessary or repeated updates, as these can generate significant historical data and increase storage usage.

    Thanks and regards,
    Anjan Kumar Chippa

  • ShivekMaharaj's avatar
    ShivekMaharaj
    Impactful Individual
    18 hours ago

    Hi JVDA98​,

    I agree with v-achippa​ that the cleanup is asynchronous, but there is one retention detail I would verify first.

    time_travel_retention_cutoff_date is not the configured retention value. Microsoft documents it as the actual earliest point from which historical data is currently available, and that can differ from the configured period.

    I would check both values with:

    SELECT
        name,
        time_travel_retention_period_days,
        time_travel_retention_cutoff_date
    FROM sys.databases;

    The supported way to configure five days is:

    ALTER DATABASE CURRENT
    SET TIME_TRAVEL_RETENTION_PERIOD = 5 DAYS;

    Microsoft covers the distinction in the Warehouse data retention documentation and the configuration guidance.

    The amount of storage can also be much larger than the current table row count suggests. Fabric Warehouse retains historical versions created by inserts, updates and deletes, so stored procedures that repeatedly update large portions of those tables can generate a substantial amount of retained Parquet data.

    Once the retention period is reduced, versions outside that window become eligible for background garbage collection, but Microsoft does not document an exact cleanup SLA. I would not manually remove the Warehouse files from OneLake.

    Your storage report also shows about 192 GB already in Soft-delete. OneLake retains soft-deleted data for seven days, and that storage is still billed during the retention window. The OneLake storage report is also cached for eight hours, so I would refresh it after the Warehouse cleanup has had time to run.

    If time_travel_retention_period_days definitely shows 5, the capacity has remained active, and files well beyond that retention window continue to remain after background cleanup, I think that would be worth taking to Microsoft Support rather than deleting anything manually.

    One other limitation to keep in mind is that the retention setting is currently Warehouse-wide, not configurable per table.