Forum Discussion
Critical section to prevent concurrent cell execution on notebook in Fabric
Hi dbeavon3
Very interesting discussion.
I haven't implemented any such thing in Fabric yet. But I agree with your approach as it can be implemented in Fabric itself.to ensure only one notebook at a time can enter a short critical section is to use the underlying OneLake (ADLS) support for leases on a file. When a notebook cell begins, it attempts to acquire an exclusive lease on a “lock file.” If the lease is granted, the notebook proceeds with the five‐second operation; otherwise, it retries or waits until the lease is released. Because OneLake runs on ADLS Gen2, this pessimistic concurrency pattern is directly supported. You can programmatically request and release the lease (for example, via Python or Spark code in the notebook), and no second notebook can acquire the same lease until it’s relinquished or expires.
No built-in Fabric library currently provides a simpler, fully internal locking mechanism across multiple Spark sessions. High concurrency mode in notebooks allows multiple sessions to share the same Spark application, but it does not itself prevent simultaneous entry into a particular code cell.
Thanks
Hi dbeavon3
Very interesting discussion.
I haven't implemented any such thing in Fabric yet. But I agree with your approach as it can be implemented in Fabric itself.
to ensure only one notebook at a time can enter a short critical section is to use the underlying OneLake (ADLS) support for leases on a file. When a notebook cell begins, it attempts to acquire an exclusive lease on a “lock file.” If the lease is granted, the notebook proceeds with the five‐second operation; otherwise, it retries or waits until the lease is released. Because OneLake runs on ADLS Gen2, this pessimistic concurrency pattern is directly supported. You can programmatically request and release the lease (for example, via Python or Spark code in the notebook), and no second notebook can acquire the same lease until it’s relinquished or expires.
No built-in Fabric library currently provides a simpler, fully internal locking mechanism across multiple Spark sessions. High concurrency mode in notebooks allows multiple sessions to share the same Spark application, but it does not itself prevent simultaneous entry into a particular code cell.
Thanks