Users within the same workspace can utilize the same launched Spark Cluster. This eliminates the need for users to wait for the startup time of Spark, and also prevents excessive CU consumption caused by launching multiple clusters. This is different from the Hi-concurrency pool, which aims to reuse clusters within the same user.
4 Comments
This has been the problem in Synapse and Fabric has the same problem.
We are running quite a lot of PySpark ETL notebooks but most of them are pretty much lightweight python scripts.
Each one of them spawns own Spark Session in Pipeline.
It would be great that Pipeline Notebooks can share Spark Sessions like the Hi-Concurrency ones for the interactive notebooks.
- Thank you for sharing this idea. We do have this in our roadmap of extending the sharing scope across multiple users as part of High Concurrency mode.
- Status added:Under Review
psanchezalvialAdvocate I
High concurrency is useful. But It would be better an approach like Databricks when you can start a cluster on their own instead of start it from a job or from a notebook.
