Forum Discussion
High Concurrency mode and custom environment - notebook sessions not shared
Hi mr_001,
Thank for reaching out in Microsoft Community Forum.
Custom environments may cause delays and disrupt session sharing, even when high concurrency is enabled.
Please follow below steps to acheive the error;1. Run a simple job, like a basic Spark command, before starting the pipeline to initialize the cluster and Spark session. This will help reduce the startup time.
2. Use high-concurrency clusters with autoscaling and preload libraries through init scripts or cluster configuration to ensure faster session initialization.
3. Set the same session tag for all notebooks and ensure spark.databricks.session.share is enabled in the Spark configuration for proper session sharing.
4. Check cluster and job logs for delays from library installation or executor setup, and preinstall dependencies or optimize custom libraries to reduce runtime delays.
If you found this post helpful, please consider marking it as "Accept as Solution" and select "Yes" if it was helpful. help other members find it more easily.
Thank you,
Pavan.
Hi mr_001,
Thank for reaching out in Microsoft Community Forum.
Custom environments may cause delays and disrupt session sharing, even when high concurrency is enabled.
Please follow below steps to acheive the error;
1. Run a simple job, like a basic Spark command, before starting the pipeline to initialize the cluster and Spark session. This will help reduce the startup time.
2. Use high-concurrency clusters with autoscaling and preload libraries through init scripts or cluster configuration to ensure faster session initialization.
3. Set the same session tag for all notebooks and ensure spark.databricks.session.share is enabled in the Spark configuration for proper session sharing.
4. Check cluster and job logs for delays from library installation or executor setup, and preinstall dependencies or optimize custom libraries to reduce runtime delays.
If you found this post helpful, please consider marking it as "Accept as Solution" and select "Yes" if it was helpful. help other members find it more easily.
Thank you,
Pavan.
Hi Pavan, I have a similar issue. All of the notebooks are attached to a custom env. I have a simple notebook to start a concurrent session. Followed by few other notebooks configured to use the same session tag but concurrent not always works. Sometimes, it fails with "Failed to create session for executing notebook".
Can you please elobrate your step #3?
3. Set the same session tag for all notebooks and ensure spark.databricks.session.share is enabled in the Spark configuration for proper session sharing.
Also, I understood that if the concurrent session doesn't exist then it should create a new instead of failing. Please share your thougths.