Forum Discussion
Sudden concurrency errors when running Fabric notebooks for ETL workloads
Most Valuable ProfessionalHi tayloramy
We already have implemented retry, even with jitter for up to 30 seconds and still face this.
We receive a lot of files, and have no control over when they arrive - we load when they arrive.
In a separate fabric instance we even face this with files that are not writing in the same table 🤔🤯
Again, we used to be able to have 10 - 20 files runing in parallel, but are now down to 1. (F64)
Super UserIf you encounter this when files are writing to different tables then somethig weird is going on.
Here's a maybe gross solution - are you able to add a source filename column (provides your source files each have a unique name) or a combination if filename+load datetime (if they do not have unique names) and then partition that table by this column? That will ensure that each file loads into it's own parquet file.
Not great for performance, but down the line you can make a copy of this table and optimize it before using it for anything.