Forum Discussion
Dataflow gen2 - Incremental refresh
Hi msprog,
Incremental refresh isn’t supported when the Dataflow Gen2 output goes straight to a Lakehouse. Right now, the supported destinations for incremental refresh are:
- Fabric Warehouse
- Fabric SQL endpoint
- Azure SQL Database
So if you need incremental refresh, you’ll need to land the data in one of those targets instead of directly in the Lakehouse.
Workarounds:
- Keep loading into the Lakehouse, but implement your own “delta logic” in Power Query or pipelines (e.g. filter on modified date and append only new rows).
- Or, load into a Warehouse with incremental refresh enabled, and if needed, create a shortcut from that Warehouse back to the Lakehouse.
If this helps, consider marking the post as solution.
Best regards!
Hi msprog,
Incremental refresh isn’t supported when the Dataflow Gen2 output goes straight to a Lakehouse. Right now, the supported destinations for incremental refresh are:
- Fabric Warehouse
- Fabric SQL endpoint
- Azure SQL Database
So if you need incremental refresh, you’ll need to land the data in one of those targets instead of directly in the Lakehouse.
Workarounds:
- Keep loading into the Lakehouse, but implement your own “delta logic” in Power Query or pipelines (e.g. filter on modified date and append only new rows).
- Or, load into a Warehouse with incremental refresh enabled, and if needed, create a shortcut from that Warehouse back to the Lakehouse.
If this helps, consider marking the post as solution.
Best regards!
Hey thanks for this. But the document says its in preview? https://learn.microsoft.com/en-us/fabric/data-factory/dataflow-gen2-incremental-refresh
