High Concurrency mode and custom environment - notebook sessions not shared

Hi mr_001,

Thank for reaching out in Microsoft Community Forum.

Custom environments may cause delays and disrupt session sharing, even when high concurrency is enabled.

Please follow below steps to acheive the error;

1. Run a simple job, like a basic Spark command, before starting the pipeline to initialize the cluster and Spark session. This will help reduce the startup time.

2. Use high-concurrency clusters with autoscaling and preload libraries through init scripts or cluster configuration to ensure faster session initialization.

3. Set the same session tag for all notebooks and ensure spark.databricks.session.share is enabled in the Spark configuration for proper session sharing.

4. Check cluster and job logs for delays from library installation or executor setup, and preinstall dependencies or optimize custom libraries to reduce runtime delays.

If you found this post helpful, please consider marking it as "Accept as Solution" and select "Yes" if it was helpful. help other members find it more easily.

Thank you,
Pavan.