Forum Discussion

kiltNone's avatar
kiltNone
New Member
6 months ago
Solved

Dataflow Publish Failed (gen 2)

I am creating a dataflow gen 2 that reads data from a data source and write it a Lakehouse in the same Fabric workspace. I works fine if I don't set incremental refresh but once I set incremental ref...
Dataflow
Need Help
  • kiltNone's avatar
    kiltNone
    6 months ago

    Hi, I manage to make it work by removing comments in the query and streamline the query.

    I set incemental refresh first once successful, I add the old data running one time with append.

     

    Thank you!

v-echaithra's avatar
v-echaithra
Community Support
6 months ago

Hi kiltNone ,

Thank you for the response.

Although Incremental Refresh is listed as supported for Lakehouse destinations, it currently works under replace-only behavior where the Dataflow must have full ownership and control of how the table is created and maintained.

If Fabric cannot guarantee that level of control during validation for example, when the table lifecycle or write pattern falls outside this tightly managed model, the publish step can fail with the generic data-destination error you’re seeing. This typically happens in scenarios beyond the simple ingestion pattern described in the documentation.

For Lakehouse-based engineering workloads, we recommend implementing incremental processing downstream using Spark (for example, MERGE operations in a Notebook or Pipeline) rather than relying on Dataflow Gen2 Incremental Refresh.

Best Regards
Chaithra E.