Forum Discussion
Critical section to prevent concurrent cell execution on notebook in Fabric
- 1 year ago
Hi dbeavon3
Very interesting discussion.
I haven't implemented any such thing in Fabric yet. But I agree with your approach as it can be implemented in Fabric itself.to ensure only one notebook at a time can enter a short critical section is to use the underlying OneLake (ADLS) support for leases on a file. When a notebook cell begins, it attempts to acquire an exclusive lease on a “lock file.” If the lease is granted, the notebook proceeds with the five‐second operation; otherwise, it retries or waits until the lease is released. Because OneLake runs on ADLS Gen2, this pessimistic concurrency pattern is directly supported. You can programmatically request and release the lease (for example, via Python or Spark code in the notebook), and no second notebook can acquire the same lease until it’s relinquished or expires.
No built-in Fabric library currently provides a simpler, fully internal locking mechanism across multiple Spark sessions. High concurrency mode in notebooks allows multiple sessions to share the same Spark application, but it does not itself prevent simultaneous entry into a particular code cell.
Thanks
Thanks, I have already used this approach in the past. I just wanted to make sure there wasn't something more obvious.
In general I do not really like using the file system for concerns that are not file-related. A lock in a database would be a LOT more intuitive than a lease in a file system. However, the LH file system is right at our fingertips, so I think it is the best option.
Ideally they would add some related functionality to "notebookutils" one day. (... eg. named mutexes or something like that).