Forum Discussion
Solved
Dataflow gen2 - Incremental refresh
Hello, I am trying Dataflow Gen 2 with incremental refresh. I read a SQL table via gateway and populate a lakehouse table. There are no transofrmations. If i run the dataflow without any incr...
Hi msprog,
Incremental refresh isn’t supported when the Dataflow Gen2 output goes straight to a Lakehouse. Right now, the supported destinations for incremental refresh are:
- Fabric Warehouse
- Fabric SQL endpoint
- Azure SQL Database
So if you need incremental refresh, you’ll need to land the data in one of those targets instead of directly in the Lakehouse.
Workarounds:
- Keep loading into the Lakehouse, but implement your own “delta logic” in Power Query or pipelines (e.g. filter on modified date and append only new rows).
- Or, load into a Warehouse with incremental refresh enabled, and if needed, create a shortcut from that Warehouse back to the Lakehouse.
If this helps, consider marking the post as solution.
Best regards!
Hi msprog ,
Did you get a chance to review the note about Lakehouse incremental refresh being in preview? For now, GA destinations are the safe bet. Curious if you’ve considered one of those as a temporary workaround until Lakehouse support matures.
Regards,
Akhil.