Forum Discussion
Delta ConcurrentAppendException when loading different tables in Lakehouse, but at the same time
- 1 year ago
I am an idiot. After further debugging i found the issue, and it was all on my side. So Fabric/Lakehouse works as intended in this case. Thank you for helping me, and sorry for wasting your time. :'(
(the conflict appears in a shared/log table, that I did not think of being a potential culprit..)
I am getting similar error "ConcurrentAppendException" when trying to update delta tables from lakehouse using parallel operations. Queries from parallel operations very simple like "update table set status = {value} where id ={id}" - where id is different in every operation. But still it is failing with error ConcurrentAppendException. I am loogin through various forums and found suggestions like -
1. retry mechanism - this is not feasible as on every transaction I can't add this code
2. batch updates - not in scope as part of parallely executing pipeline I am running these notebooks with update queries
3. Partitioning - for partitioning id column is primary key column so it will create lot of files
4. Isolation level - This is not sure but I think, In Microsoft Fabric, Delta tables in the Lakehouse use Snapshot Isolation by default. But I haven't found any way to change this to serializable.
5. In delta.io they are indicating to use more specific filter as per partition but in my case I don't have much data in table and also no other column to make it specific filter. As I am directly doing it on primary key
Seems like very basic requirement in Data engineering but struggling to achieve it in Microsoft Fabric. Do we have any solution on this?