Notebook concurrency in data pipelines not working

Hi PEZI ,

Thank you for reaching out to fabric community!!

Based on the Microsoft Fabric documentation, session reuse is not guaranteed simply because two notebooks have the same session tag. The Azure Data Factory Pipeline activity between the notebooks can prevent the second notebook from attaching to the original Spark session.

Microsoft documents that High Concurrency session reuse works only when the notebook activity can attach to an existing compatible High Concurrency session. For a session to be reused, all of the following must match:

• Same user
• Same workspace
• Same default Lakehouse
• Same Spark compute configuration
• Same libraries
• Same session tag (for pipeline notebook activities)
• An existing High Concurrency session must still be available to attach to.
  
  

https://learn.microsoft.com/en-us/fabric/data-engineering/configure-high-concurrency-session-notebooks-in-pipelines#session-sharing-conditions 

In your first test , notebook 1 and the second notebook immediately follows the first notebook, so Fabric can attach to the existing High Concurrency session.

In your second test , notebook 1 -> Execute Azure Data Factory Pipeline -> Notebook 2 , the pipeline execution introduces another activity between the notebooks. Microsoft documentation does not state that High Concurrency sessions are preserved across non-notebook pipeline activities. High Concurrency for pipelines is specifically designed for Notebook activities sharing the same session tag.

Thanks!!