Forum Discussion
Delta ConcurrentAppendException when loading different tables in Lakehouse, but at the same time
I am an idiot. After further debugging i found the issue, and it was all on my side. So Fabric/Lakehouse works as intended in this case. Thank you for helping me, and sorry for wasting your time. :'(
(the conflict appears in a shared/log table, that I did not think of being a potential culprit..)
Hello FelixL
Fabric Lakehouse uses a queuing system where concurrency depends on the capacity SKU (e.g., F2 allows 1 concurrent job; F32 allows 8).
If your capacity tier is too low, concurrent writes to any table in the Lakehouse may trigger conflicts, as jobs compete for limited resources
For 10 concurrent jobs, use at least F16 (burst factor 3 → 32×3 = 96 cores).
Upgrade SKUs to match workload parallelism requirements using Microsoft’s formula
Required SKU Tier = Ceil(Total Concurrent Jobs / Burst Factor)
https://learn.microsoft.com/en-us/fabric/data-engineering/spark-job-concurrency-and-queueing
Try partition by natural keys to prevent overlapping file operations
Thanks for your reply, but I think we are talking sbout different things here..?
I am running one single spark session, using 12 vcores, in a F64/P1 capacity. This one small session then writes to 80 tables in my lakehouse, using a "run multiple" notebooks concurrency of 3.
I should have enormous compute overhead - and I am not seeing any concurrency throttling or job queuing.What I am seeing are error messages saying that jobs try to write to delta tables they are not actually touching. This either needs a good explanation, or appears to be some bug in the lakehouse-onelake sync layer..?
Makes sesnse now, its not a capacity issue. Let me dig down further. My gut feeling says it is related to , but let me come back on this
"isolationLevel":"Serializable"
I am an idiot. After further debugging i found the issue, and it was all on my side. So Fabric/Lakehouse works as intended in this case. Thank you for helping me, and sorry for wasting your time. :'(
(the conflict appears in a shared/log table, that I did not think of being a potential culprit..)