Users within the same workspace can utilize the same launched Spark Cluster. This eliminates the need for users to wait for the startup time of Spark, and also prevents excessive CU consumption caus...
This has been the problem in Synapse and Fabric has the same problem.
We are running quite a lot of PySpark ETL notebooks but most of them are pretty much lightweight python scripts.
Each one of them spawns own Spark Session in Pipeline.
It would be great that Pipeline Notebooks can share Spark Sessions like the Hi-Concurrency ones for the interactive notebooks.
