Forum Discussion
Dataflow Gen2 error in Fabric
Hi Anonymous,
Thanks for sharing the update. The error you are seeing "Lakehouse036: conflicting metadata change" is happening because multiple Dataflow Gen2 jobs are writing to the same Lakehouse table in parallel. When this happens, each dataflow tries to commit changes to the table metadata, which leads to conflicts, and some of them fail.
To resolve your issue try avoiding parallel writes to the same table. Instead of 5 Dataflows writing into the same final table at the same time, run them sequentially. This will ensure there are no metadata conflicts when committing changes.
Or you can use staging tables. Point each yearly Dataflow to its own staging table in the Lakehouse (e.g., Table_2019, Table_2020, etc.). After that, you can either use a notebook or Pipeline activity to merge these staging tables into your final table, or create a lakehouse shortcut or union query on top of them.
If you prefer keeping the Dataflows, orchestrate them using a Fabric Data Pipeline to run one after another (or write to staging first, then merge).
This approach removes the metadata conflict and will make your loads more stable.
Best Regards,
Hammad.
But I am also seeing on the internet that its timeout is 8hrs. Anonymous Can you please help on this?
Thanks
Hi Anonymous,
Yes right the execution time limit is up to 8 hours, depending on factors like workload, SKU, and internal resource management. However, in real-world usage, especially in production environments (including with high SKUs like F64 or F128), many users have observed a hard stop at exactly 4 hours like in your case.
Also the best use case to transfer this amount of data is by spliting the load. Likje you can break the source table into smaller partitions ( like by date range, primary key buckets, or any natural partitioning column). Then create multiple parameterized dataflows or loops using pipeline to load each partition individually and avoid hitting the execution time limit.
You can also use Data Pipelines with Copy Activity. Since you're only copying data without transformations, using Copy Activity in Fabric Data Pipelines from Databricks to Lakehouse can be more efficient and does not have execution time restriction. It will also handles larger datasets more gracefully.
If I misunderstand your needs or you still have problems on it, please feel free to let us know.
Best Regards,
Hammad.