Forum Discussion
Dataflow Gen2 error in Fabric
Hi Anonymous,
Thanks for sharing the update. The error you are seeing "Lakehouse036: conflicting metadata change" is happening because multiple Dataflow Gen2 jobs are writing to the same Lakehouse table in parallel. When this happens, each dataflow tries to commit changes to the table metadata, which leads to conflicts, and some of them fail.
To resolve your issue try avoiding parallel writes to the same table. Instead of 5 Dataflows writing into the same final table at the same time, run them sequentially. This will ensure there are no metadata conflicts when committing changes.
Or you can use staging tables. Point each yearly Dataflow to its own staging table in the Lakehouse (e.g., Table_2019, Table_2020, etc.). After that, you can either use a notebook or Pipeline activity to merge these staging tables into your final table, or create a lakehouse shortcut or union query on top of them.
If you prefer keeping the Dataflows, orchestrate them using a Fabric Data Pipeline to run one after another (or write to staging first, then merge).
This approach removes the metadata conflict and will make your loads more stable.
Best Regards,
Hammad.
In addition to this, I have divided the data according to the year, and I am using 5 parallel Dataflows now and targeting them to one Final table in Lakehouse. While trying this, some of my Dataflows failed, and I am getting a Dataflow Gen2 error.
semantic_table: There was a problem refreshing the dataflow: 'Couldn't refresh the entity because of an issue with the mashup document MashupException.Error: Error in comitting version., Underlying error: conflicting metadata change Details: Reason = DataSource.Error;ErrorCode = Lakehouse036;Message = conflicting metadata change;Message.Format = conflicting metadata change;Microsoft.Data.Mashup.Error.Context = User GatewayObjectId: e58a7629-e2e0-412b-8688-6f0ebc4f1e50'. Error code: 104100. (Request ID: 3cbf7621-0a97-481e-be25-97ad4ab717f6).
Can you please let me know about this part? And what will be the solution for this?
Thanks!
Hi Anonymous,
Thanks for sharing the update. The error you are seeing "Lakehouse036: conflicting metadata change" is happening because multiple Dataflow Gen2 jobs are writing to the same Lakehouse table in parallel. When this happens, each dataflow tries to commit changes to the table metadata, which leads to conflicts, and some of them fail.
To resolve your issue try avoiding parallel writes to the same table. Instead of 5 Dataflows writing into the same final table at the same time, run them sequentially. This will ensure there are no metadata conflicts when committing changes.
Or you can use staging tables. Point each yearly Dataflow to its own staging table in the Lakehouse (e.g., Table_2019, Table_2020, etc.). After that, you can either use a notebook or Pipeline activity to merge these staging tables into your final table, or create a lakehouse shortcut or union query on top of them.
If you prefer keeping the Dataflows, orchestrate them using a Fabric Data Pipeline to run one after another (or write to staging first, then merge).
This approach removes the metadata conflict and will make your loads more stable.
Best Regards,
Hammad.
Hi Anonymous,
As we haven’t heard back from you, so just following up to our previous message. I'd like to confirm if you've successfully resolved this issue or if you need further help.
If yes, you are welcome to share your workaround so that other users can benefit as well. And if you're still looking for guidance, feel free to give us an update, we’re here for you.
Best Regards,
Hammad.
Hi Anonymous, Thanks for helping. The issue is resolved