I’m loading a large fact table (40–50M rows) into a Fabric Warehouse using Dataflows Gen2

[RamReddy09]

I’m loading a large fact table (40–50M rows) into a Fabric Warehouse using Dataflows Gen2. The load works, but incremental refresh isn’t kicking in. Every run is doing a full reload even though I enabled incremental settings. Am I missing something?

[Ugk161610 · Accepted solution]

Hi @RamReddy09 ,

I ran into the same issue last month. In Dataflows Gen2, incremental refresh only works if the column you select for the ‘modified date’ is actually changing in the source.

In my case the timestamp column looked fine but wasn’t updating for late-arriving rows, so Fabric treated every row as new and reloaded everything.

What fixed it for me:

• I switched the incremental column to the actual LastUpdated field from the source.

• Cleared the staging table in the Warehouse (it still had the previous full load metadata).

• Re-published the Dataflow and triggered a fresh run.

After that, only the changed rows got processed and the load time dropped a lot.

Just make sure the column you use really reflects row-level changes — Fabric relies on that to decide the delta.

GopiKrishna

[tayloramy]

Hi @RamReddy09, 

Can you share some screenshots of your dataflow gen 2 set up? 

If you found this helpful, consider giving some Kudos. If I answered your question or solved your problem, mark this post as the solution.