Forum Discussion

kiltNone's avatar
kiltNone
New Member
6 months ago
Solved

Dataflow Publish Failed (gen 2)

I am creating a dataflow gen 2 that reads data from a data source and write it a Lakehouse in the same Fabric workspace. I works fine if I don't set incremental refresh but once I set incremental ref...
Dataflow
Need Help
  • kiltNone's avatar
    kiltNone
    6 months ago

    Hi, I manage to make it work by removing comments in the query and streamline the query.

    I set incemental refresh first once successful, I add the old data running one time with append.

     

    Thank you!

v-echaithra's avatar
v-echaithra
Icon for Community Support rankCommunity Support
6 months ago

Hi kiltNone ,

Glad to hear you were able to get it working.

Streamlining the query and removing inline comments can help ensure that the query definition validates cleanly and preserves full query folding, which is required for Incremental Refresh to be accepted during publishing.
Configuring Incremental Refresh first allows the dataflow to create and take ownership of the destination table with the expected metadata and structure.
Your approach is valid and aligns with the recommended guidance Incremental Refresh must establish and own the table initially, which you enabled before loading the historical data. The one-time append functions as an initial backfill, while subsequent refreshes continue to follow the replace only behavior managed by the dataflow.


Reference: Incremental refresh in Dataflow Gen2 - Microsoft Fabric | Microsoft Learn
If further assistance is still required. We are available to support you and are committed to helping you reach a resolution.


Thank you.