Async merge risk

Hi smpa01 ,

Thank you for reaching out to the Microsoft Fabric Community Forum.

When merging multiple independent tables asynchronously from Bronze (non-Lakehouse) to Silver (Lakehouse Delta), and assuming no cross-table dependencies, the pattern is generally safe but there are still some risks and practical considerations to be consider.

Even if the tables are fully independent, running multiple merge operations at the same time can cause some problems. Each merge uses system resources like CPU, memory, and input/output (IO), so if you run many merges together, especially on a smaller cluster, it can slow down performance or even cause jobs to fail.

Also, if all the merge operations are reading from or writing to the same storage system as ADLS or One Lake, it can create IO bottlenecks or hit bandwidth limits, particularly when dealing with large volumes of data. Merge operations also involve internal processing (called shuffling), which uses a lot of memory. If too many such processes run at once, it can lead to memory issues or spill data to disk, which slows things down.

Lastly, when running merges asynchronously (in parallel), it becomes harder to catch errors. If one merge fails, you may not notice it unless you specifically check or handle it in your code.

So, while this pattern is supported and commonly used, it’s important to control the number of concurrent merges, monitor your cluster performance, and handle errors properly to avoid performance or reliability issues.

Hope this helps. Please reach out for further assistance.

Thank you.