Sudden concurrency errors when running Fabric notebooks for ETL workloads

Hi MirjamPD,

I’ve seen this exact error when multiple Spark jobs try to write to the same Delta table (or the same partitions) at roughly the same time. Delta Lake uses optimistic concurrency, if another transaction adds files that overlap what your job is reading/writing, Spark throws ConcurrentAppendException rather than silently merging the changes. 

I don't think there's been a recent change here - I experienced this issue many months ago. In your notebooks you could try to catch the ConcurrentAppendException error, and then pause for 5 seconds and try again. This isn't a great solution, but it's an easy one to implement. 

A more robust solution would be for your concurrent notebooks to each write to their own table and then after all notebooks are complete, have a last step notebook that merges the data all into one table. 

If you found this helpful, consider giving some Kudos. If I answered your question or solved your problem, mark this post as the solution.
Taylor Amy.