Forum Discussion
Notebook concurrency in data pipelines not working
Hi all,
I have a fabric data pipeline that runs a spark notebook then a pipeline task that calls a Azure Data Factory Pipeline and then another notebook. High concurrency is enabled, notebooks share the same environment, have the same session tag, timeout is set to multiple hours etc. I tested first the run by deactivating the pipeline task, so only the 2 notebooks run and they share the same Spark cluster correctly:
Then I made a full run and for some reason the second notebook is not reusing the first cluster, but starts another one. Can somebody explain if this is a bug due connected to the pipeline task that calls an Azure Data Factory?
Thanks for your help,
Peter
Hi PEZI ,
Thank you for reaching out to fabric community!!
Based on the Microsoft Fabric documentation, session reuse is not guaranteed simply because two notebooks have the same session tag. The Azure Data Factory Pipeline activity between the notebooks can prevent the second notebook from attaching to the original Spark session.
Microsoft documents that High Concurrency session reuse works only when the notebook activity can attach to an existing compatible High Concurrency session. For a session to be reused, all of the following must match:
- Same user
- Same workspace
- Same default Lakehouse
- Same Spark compute configuration
- Same libraries
- Same session tag (for pipeline notebook activities)
- An existing High Concurrency session must still be available to attach to.
In your first test , notebook 1 and the second notebook immediately follows the first notebook, so Fabric can attach to the existing High Concurrency session.In your second test , notebook 1 -> Execute Azure Data Factory Pipeline -> Notebook 2 , the pipeline execution introduces another activity between the notebooks. Microsoft documentation does not state that High Concurrency sessions are preserved across non-notebook pipeline activities. High Concurrency for pipelines is specifically designed for Notebook activities sharing the same session tag.
Thanks!!
3 Replies
v-sathmakuriCommunity Support
Hi PEZI ,
Thank you for reaching out to fabric community!!
Based on the Microsoft Fabric documentation, session reuse is not guaranteed simply because two notebooks have the same session tag. The Azure Data Factory Pipeline activity between the notebooks can prevent the second notebook from attaching to the original Spark session.
Microsoft documents that High Concurrency session reuse works only when the notebook activity can attach to an existing compatible High Concurrency session. For a session to be reused, all of the following must match:
- Same user
- Same workspace
- Same default Lakehouse
- Same Spark compute configuration
- Same libraries
- Same session tag (for pipeline notebook activities)
- An existing High Concurrency session must still be available to attach to.
In your first test , notebook 1 and the second notebook immediately follows the first notebook, so Fabric can attach to the existing High Concurrency session.In your second test , notebook 1 -> Execute Azure Data Factory Pipeline -> Notebook 2 , the pipeline execution introduces another activity between the notebooks. Microsoft documentation does not state that High Concurrency sessions are preserved across non-notebook pipeline activities. High Concurrency for pipelines is specifically designed for Notebook activities sharing the same session tag.
Thanks!!
Hi v-sathmakuri ,
thanks for the review. I agree with you, I think that intermediate activity between the notebooks closes the active spark session as the pipeline leaves the execution context (starting an ADF pipeline). Not sure, if that would be the case for all activites (e.g. wait-activity) but as I am starting a completly different Azure Service that´s probably the reason. I will try to trigger the ADF pipeline via API from a Fabric notebook, hopefully this will preserve the active spark session.
Thanks,
Peter"First, We verify that the pipeline is configured to run notebooks in parallel and not sequentially. Then, we check whether the Databricks cluster has sufficient resources and autoscaling enabled. we review the job logs to see if executions are queued or waiting for cluster availability. we also check for Spark scheduling issues or concurrent access to the same data. Based on the findings, we scale the cluster, adjust concurrency settings, or optimize the notebook execution to restore parallel processing."
