Forum Discussion
Spark Pool Usage Across Workspaces with Promoted Fabric Environments
Hi everyone,
We have a centralized workspace in Microsoft Fabric where we manage and promote Environment items (e.g., Spark configurations). Multiple other workspaces and notebooks connect to these shared environments.
I’d like to understand how Spark compute is handled in this setup:
- Do notebooks across different workspaces share the same underlying Spark pool/resources when connected to a centralized Environment?
- Or does each notebook/session spin up its own separate Spark instance, even if they reference the same Environment?
In short, is the compute truly shared, or isolated per notebook/session despite using a common Environment?
Thanks in advance for your insights!
Hi AdarshChekodu ,
Thanks for reaching out to the Microsoft Fabric Community forum.In Microsoft Fabric, an Environment item is primarily a configuration and dependency management layer, not a shared Spark compute pool. Even when multiple notebooks across different workspaces reference the same centralized Environment, the actual Spark compute is still provisioned at the notebook/session level.
This means that:
- The shared Environment helps standardize settings such as:
- Spark configurations
- Libraries/packages
- Runtime dependencies
- However, it does not cause notebooks to share the same active Spark session or cluster instance.
In practice, each notebook execution typically starts or attaches to its own Spark session/runtime context, even if the notebooks are using the exact same Environment item. So the compute is generally isolated per notebook/session, while the Environment acts as a reusable configuration template across workspaces.
If your goal is to have multiple notebooks use the same spark session, there is a feature called high concurrency mode, you could attach multiple notebooks to the high concurrency session getting a spark session instantly to run the queries and achieve a greater session utilization rate.
Configure high concurrency mode for notebooks - Microsoft Fabric | Microsoft Learn
I hope this information helps. Please do let us know if you have any further queries.
Thank you- The shared Environment helps standardize settings such as:
1 Reply
Hi AdarshChekodu ,
Thanks for reaching out to the Microsoft Fabric Community forum.In Microsoft Fabric, an Environment item is primarily a configuration and dependency management layer, not a shared Spark compute pool. Even when multiple notebooks across different workspaces reference the same centralized Environment, the actual Spark compute is still provisioned at the notebook/session level.
This means that:
- The shared Environment helps standardize settings such as:
- Spark configurations
- Libraries/packages
- Runtime dependencies
- However, it does not cause notebooks to share the same active Spark session or cluster instance.
In practice, each notebook execution typically starts or attaches to its own Spark session/runtime context, even if the notebooks are using the exact same Environment item. So the compute is generally isolated per notebook/session, while the Environment acts as a reusable configuration template across workspaces.
If your goal is to have multiple notebooks use the same spark session, there is a feature called high concurrency mode, you could attach multiple notebooks to the high concurrency session getting a spark session instantly to run the queries and achieve a greater session utilization rate.
Configure high concurrency mode for notebooks - Microsoft Fabric | Microsoft Learn
I hope this information helps. Please do let us know if you have any further queries.
Thank you- The shared Environment helps standardize settings such as: