Forum Discussion
Dataflow Publish Failed (gen 2)
Hi, I manage to make it work by removing comments in the query and streamline the query.
I set incemental refresh first once successful, I add the old data running one time with append.
Thank you!
Community SupportHi kiltNone ,
Thank you for reaching out to Microsoft Community.
The failure is not caused by configuration error alone, but by how Dataflow Gen2 currently implements Incremental Refresh with Lakehouse:
Incremental extraction is supported but converting an existing table to incremental is not.
Destination writes are full replacements, not merges. Validation gaps surface as a generic publishing error.
If the Lakehouse table was originally created by a non-incremental load, Fabric cannot convert it into an incremental table when IR is later enabled.The destination must be explicitly configured (Lakehouse, schema mode, and mapping must be defined before publishing).
Even when Incremental Refresh is enabled, Dataflow Gen2 rewrites the destination table during refresh.
It does not support transactional append/upsert semantics like Spark or Warehouse pipelines.
Recommended Workarounds:
Recreate the Destination Table
You must start with a clean table created by the incremental-enabled dataflow. Ensure Incremental Refresh is configured. Publish the Dataflow again so Fabric creates the table with the correct metadata. Do not enable IR on top of an already-loaded table. Ensure the Incremental Filter Is the First Transformation.
Hope this helps.
Thank you.
Thanks for the advice. I tried creating a new Lakehouse and try to run with Incremental Refresh configure from the beginning. I still encounter the same error.