Forum Discussion
Dataflow gen2 - Incremental refresh
Hello,
I am trying Dataflow Gen 2 with incremental refresh.
I read a SQL table via gateway and populate a lakehouse table. There are no transofrmations.
If i run the dataflow without any incremental refresh configured , it runs fine.
However, if i enable incremental refresh, it fails. the error is below.
the query script is simple:
error is:
There was a problem refreshing the dataflow: 'Couldn't refresh the entity because of an issue with the mashup document MashupException.Error: The value does not support versioning. Details: Reason = Expression.Error;ErrorCode = 10855;Detail = #table({"ProductKey", "OrderDateKey", "DueDateKey", "ShipDateKey", "CustomerKey", "PromotionKey", "CurrencyKey", "SalesTerritoryKey", "SalesOrderNumber", "SalesOrderLineNumber", "RevisionNumber", "OrderQuantity", "UnitPrice", "ExtendedAmount", "UnitPriceDiscountPct", "DiscountAmount", "ProductStandardCost", "TotalProductCost", "SalesAmount", "TaxAmt", "Freight", "CarrierTrackingNumber", "CustomerPONumber", "OrderDate", "DueDate", "ShipDate"}, {});Microsoft.Data.Mashup.Error.Context = User GatewayObjectId: a0715ee3-xxxx-4a77-912b-3bf5ebc598c0'. Error code: Mashup Exception Expression Error. (Request ID: a98070c6-6d8e-4961-88f0-xxxxxx).
Please does anyone has experience with this.
Hi msprog,
Incremental refresh isn’t supported when the Dataflow Gen2 output goes straight to a Lakehouse. Right now, the supported destinations for incremental refresh are:
- Fabric Warehouse
- Fabric SQL endpoint
- Azure SQL Database
So if you need incremental refresh, you’ll need to land the data in one of those targets instead of directly in the Lakehouse.
Workarounds:
- Keep loading into the Lakehouse, but implement your own “delta logic” in Power Query or pipelines (e.g. filter on modified date and append only new rows).
- Or, load into a Warehouse with incremental refresh enabled, and if needed, create a shortcut from that Warehouse back to the Lakehouse.
If this helps, consider marking the post as solution.
Best regards!
4 Replies
Hi msprog,
Incremental refresh isn’t supported when the Dataflow Gen2 output goes straight to a Lakehouse. Right now, the supported destinations for incremental refresh are:
- Fabric Warehouse
- Fabric SQL endpoint
- Azure SQL Database
So if you need incremental refresh, you’ll need to land the data in one of those targets instead of directly in the Lakehouse.
Workarounds:
- Keep loading into the Lakehouse, but implement your own “delta logic” in Power Query or pipelines (e.g. filter on modified date and append only new rows).
- Or, load into a Warehouse with incremental refresh enabled, and if needed, create a shortcut from that Warehouse back to the Lakehouse.
If this helps, consider marking the post as solution.
Best regards!Hey thanks for this. But the document says its in preview? https://learn.microsoft.com/en-us/fabric/data-factory/dataflow-gen2-incremental-refresh
Hi msprog ,
Documentation lists Fabric Lakehouse (preview) as supported for incremental refresh but it’s not fully functional yet.
For now, incremental refresh works reliably only with other GA destinations.
Keep an eye on Microsoft Fabric Roadmap Microsoft’s roadmap as Lakehouse incremental refresh continues to evolve toward production readiness.
Regards,
Akhil.Hi msprog ,
Did you get a chance to review the note about Lakehouse incremental refresh being in preview? For now, GA destinations are the safe bet. Curious if you’ve considered one of those as a temporary workaround until Lakehouse support matures.
Regards,
Akhil.
