DataflowsStagingLakehouse Size

Hi nicolasf​,

The main thing I would clarify is that DataflowsStagingLakehouse is an internal Dataflow Gen2 staging artifact, not a normal Lakehouse that you should manage manually.

Microsoft documents that these staging items are created and managed automatically by Dataflow Gen2 and strongly advises users not to modify, rename or delete them directly because that can cause unexpected behavior.

The staging Lakehouse stores intermediate data used during Dataflow Gen2 transformations, so if you have many dataflows with staging enabled, especially with large intermediate datasets, the storage can become significant.

Microsoft explains the staging behavior here in the Dataflow Gen2 staging documentation.

I would also be careful with this command:

ALTER DATABASE DataflowsStagingLakehouse
SET TIME_TRAVEL_RETENTION_PERIOD = 7 DAYS;

Microsoft documents TIME_TRAVEL_RETENTION_PERIOD as a Fabric Warehouse retention setting. I would not use it to try to manage the internal Dataflow Gen2 staging Lakehouse.

The safer approach is:

• leave DataflowsStagingLakehouse itself untouched
• review which Dataflows Gen2 have staging enabled
• disable staging where it is not providing a performance benefit
• use the Dataflow Gen2 performance guidance to reduce unnecessary staging
• monitor storage growth after those changes

Microsoft’s current guidance also notes that staging is not always needed. For example, when loading directly to some destinations, staging can be disabled, and it is most useful when transformations cannot fold efficiently to the source.

So I would optimize the Dataflows that generate the staging data rather than trying to clean or change retention settings on the system staging Lakehouse itself.

AI-assisted drafting: AI was used to help structure and phrase this response. I reviewed and validated the technical content before posting.

Hi nicolasf​,

The staging lakehouse is used for staging for dataflow gen 2. If you have lots of dataflows, then this staging will get rather big. 

It is strongly advised that you do not touch this lakehouse at all. 

Data in Dataflow Gen2 staging - Microsoft Fabric | Microsoft Learn

Hi nicolasf​,

Have you had a chance to review the solution shared by tayloramy​ ShivekMaharaj​ codeautomation​ ? If the issue persists, feel free to reply so we can help further.

Thank you.

HI nicolasf​,

Checking in to see if your issue has been resolved. let us know if you still need any assistance.

Thank you.

Tx tayloramy​ . But i've done this :

ALTER DATABASE DataflowsStagingLakehouse 

SET TIME_TRAVEL_RETENTION_PERIOD  = 7 DAYS;

... and the storage seems to be moved on soft delete storage.
I don't understand why this instruction had effect on a lakehouse.. I thought it was specific for warehouses

Is it possible to perform a comprehensive storage assessment of my Microsoft Fabric environment (Lakehouse and/or Warehouse) to identify the main drivers of storage consumption and explain any unusually high storage usage?

The assessment should provide:

• An overall breakdown of storage consumption across data, system files, metadata, Delta logs, version history, and other storage components.
• Identification of the largest storage consumers (tables, datasets, files, or objects).
• Analysis of the impact of retention settings, including Time Travel and historical version retention.
• Assessment of the storage impact of auditing, logging, monitoring, and governance-related features.
• Detection of potential inefficiencies or anomalies, such as excessive version accumulation, obsolete files, fragmented storage, or a large number of small files.
• Trends and factors contributing to storage growth over time.
• Recommendations and optimization opportunities to reduce storage consumption while maintaining operational and compliance requirements.

The objective is to obtain a global diagnostic of the Microsoft Fabric storage footprint, understand the root causes of storage usage, and identify opportunities for optimization.

Hi nicolasf​,

On your follow-up about a full storage assessment: the staging lakehouse itself is system-managed, but for your own Lakehouses you can build most of that breakdown with one notebook. The key split is current data vs everything else (old Delta versions, unreferenced files):

total folder size   = everything physically stored for the table
current size        = files the latest Delta version uses
total - current     = history / unvacuumed files
files vs numFiles   = small-file fragmentation

A notebook attached to the Lakehouse you want to assess:

from pyspark.sql import functions as F

def folder_size(path):
    total, n = 0, 0
    for f in notebookutils.fs.ls(path):
        if f.isDir:
            s, c = folder_size(f.path)
            total, n = total + s, n + c
        else:
            total, n = total + f.size, n + 1
    return total, n

rows = []
for t in spark.catalog.listTables():
    d = spark.sql(f"DESCRIBE DETAIL {t.name}").collect()[0]
    total_bytes, all_files = folder_size(d["location"])
    versions = spark.sql(f"DESCRIBE HISTORY {t.name}").count()
    rows.append((t.name, d["sizeInBytes"], total_bytes,
                 d["numFiles"], all_files, versions))

report = (spark.createDataFrame(rows,
            "tbl string, current_bytes long, total_bytes long, "
            "current_files long, all_files long, versions long")
    .withColumn("current_gb", F.round(F.col("current_bytes") / 1e9, 2))
    .withColumn("history_gb",
                F.round((F.col("total_bytes") - F.col("current_bytes")) / 1e9, 2))
    .withColumn("avg_file_mb",
                F.round(F.col("current_bytes") / F.col("current_files") / 1e6, 1))
    .orderBy(F.desc("total_bytes")))

display(report)

How to read the result:

history_gb high        -> frequent overwrites/merges, run VACUUM
avg_file_mb very small -> many small files, run OPTIMIZE
versions very high     -> append/merge every few minutes; batch it
current_gb high        -> genuine data volume; check columns kept

The clean-up commands, per table (keep the default 7-day retention unless you are sure you don't need time travel):

OPTIMIZE my_table;
VACUUM my_table RETAIN 168 HOURS;

For the workspace-level view (including system items like DataflowsStagingLakehouse), the Storage page of the Fabric Capacity Metrics app breaks OneLake storage down by workspace and over time. Deleted data also stays billable for the soft-delete retention period, so a drop in size can lag behind a cleanup by several days.

So my overall approach would be:

Capacity Metrics storage page -> which workspace/item grows
-> notebook report per Lakehouse -> current vs history vs small files
-> OPTIMIZE + VACUUM where history or fragmentation dominates
-> rerun monthly to track growth trends

For the staging lakehouse specifically, the earlier advice still applies: don't touch it directly, and reduce it by disabling staging on dataflows that don't need it.

AI-assisted drafting: AI was used to help structure and phrase this response. I reviewed and validated the technical content before posting.