Forum Discussion

dbeavon3's avatar
dbeavon3
Memorable Member
1 year ago
Solved

Critical section to prevent concurrent cell execution on notebook in Fabric

We have some resources in Fabric that need to be protected from concurrent access.  Assume there are 10 notebooks running at the exact same time for 30 mins.  Each notebook runs on its own session, b...
  • nilendraFabric's avatar
    1 year ago

    Hi dbeavon3 

     

    Very interesting discussion. 
    I haven't implemented any such thing in Fabric yet. But I agree with your approach as it can be implemented in Fabric itself.

     

    to ensure only one notebook at a time can enter a short critical section is to use the underlying OneLake (ADLS) support for leases on a file. When a notebook cell begins, it attempts to acquire an exclusive lease on a “lock file.” If the lease is granted, the notebook proceeds with the five‐second operation; otherwise, it retries or waits until the lease is released. Because OneLake runs on ADLS Gen2, this pessimistic concurrency pattern is directly supported. You can programmatically request and release the lease (for example, via Python or Spark code in the notebook), and no second notebook can acquire the same lease until it’s relinquished or expires.

    https://learn.microsoft.com/en-us/azure/storage/blobs/concurrency-manage?toc=%2Fazure%2Fstorage%2Fblobs%2Ftoc.json

    No built-in Fabric library currently provides a simpler, fully internal locking mechanism across multiple Spark sessions. High concurrency mode in notebooks allows multiple sessions to share the same Spark application, but it does not itself prevent simultaneous entry into a particular code cell.

     

     

    Thanks