Forum Discussion
Spark Pool Usage Across Workspaces with Promoted Fabric Environments
- 3 months ago
Hi AdarshChekodu ,
Thanks for reaching out to the Microsoft Fabric Community forum.In Microsoft Fabric, an Environment item is primarily a configuration and dependency management layer, not a shared Spark compute pool. Even when multiple notebooks across different workspaces reference the same centralized Environment, the actual Spark compute is still provisioned at the notebook/session level.
This means that:
- The shared Environment helps standardize settings such as:
- Spark configurations
- Libraries/packages
- Runtime dependencies
- However, it does not cause notebooks to share the same active Spark session or cluster instance.
In practice, each notebook execution typically starts or attaches to its own Spark session/runtime context, even if the notebooks are using the exact same Environment item. So the compute is generally isolated per notebook/session, while the Environment acts as a reusable configuration template across workspaces.
If your goal is to have multiple notebooks use the same spark session, there is a feature called high concurrency mode, you could attach multiple notebooks to the high concurrency session getting a spark session instantly to run the queries and achieve a greater session utilization rate.
Configure high concurrency mode for notebooks - Microsoft Fabric | Microsoft Learn
I hope this information helps. Please do let us know if you have any further queries.
Thank you - The shared Environment helps standardize settings such as:
Hi AdarshChekodu ,
Thanks for reaching out to the Microsoft Fabric Community forum.
In Microsoft Fabric, an Environment item is primarily a configuration and dependency management layer, not a shared Spark compute pool. Even when multiple notebooks across different workspaces reference the same centralized Environment, the actual Spark compute is still provisioned at the notebook/session level.
This means that:
- The shared Environment helps standardize settings such as:
- Spark configurations
- Libraries/packages
- Runtime dependencies
- However, it does not cause notebooks to share the same active Spark session or cluster instance.
In practice, each notebook execution typically starts or attaches to its own Spark session/runtime context, even if the notebooks are using the exact same Environment item. So the compute is generally isolated per notebook/session, while the Environment acts as a reusable configuration template across workspaces.
If your goal is to have multiple notebooks use the same spark session, there is a feature called high concurrency mode, you could attach multiple notebooks to the high concurrency session getting a spark session instantly to run the queries and achieve a greater session utilization rate.
Configure high concurrency mode for notebooks - Microsoft Fabric | Microsoft Learn
I hope this information helps. Please do let us know if you have any further queries.
Thank you