Environment for Python packages to be used by different PySpark scripts in different Workspaces

[tan_thiamhuat]

let us say I have created a new Environment named Environment_A for all the different Python packages I need for the PySpark script to run inside Workspace_A. Can this Environment_A be used for another PySpark script running in Workspace_B? I find at times, it works, and other times, it gets disconnected instantly.

The question then is how can I create a new Environment to be used in different workspaces for different PySpark scripts? You know create a new environment with the different Python packages takes time, and we do not want to duplicate the same environment all over the different workspaces.

What is the best solution then?

[tayloramy]

Hi @tan_thiamhuat, 

If you want Microsoft to look into it, you will need to open a support ticket. These forums are community forums, not an official support channel. 

From what I can tell, you're doing things correctly, so opening a ticket would be the correct move at this point.

If you found this helpful, consider giving some Kudos. If I answered your question or solved your problem, please mark this as the solution.

[v-hashadapu]

Hi @tan_thiamhuat , Thank you for reaching out to the Microsoft Community Forum.

Environments in Microsoft Fabric are inherently tied to the workspace they are created in and cannot be directly shared as standalone environments across different workspaces. However, if multiple workspaces are attached to the same Fabric capacity, like within a trial capacity, Fabric may reuse cached environment containers behind the scenes. This makes environment sharing appear to work occasionally but it is not a supported or reliable feature and can lead to random disconnections.

You cannot create an environment outside any workspace, as all Fabric items must reside within workspaces attached to a capacity. The best practice when you need consistent Python environments across multiple workspaces is to automate environment creation using shared package configuration files (requirements.txt, conda.yml) or consider containerizing environments with Azure Container Registry for guaranteed consistency. This ensures reproducibility without duplicating effort or relying on unstable cache mechanisms. Planning proper environment management early is key, especially when moving from trial to dedicated capacities.

Workspaces in Microsoft Fabric and Power BI - Microsoft Fabric | Microsoft Learn

Create, Configure, and Use an Environment in Fabric - Microsoft Fabric | Microsoft Learn

[tan_thiamhuat]

yes, becauuse I have only 1 trial capacity.

[tayloramy]

If everything is on the same capacity, workspaces have the same network settings, and users all have pro licenses, then I'm not sure what else to suggest. Based on the documentation, that is all you need to do. It might be time to open a ticket and see what Microsoft says.

If you found this helpful, consider giving some Kudos. If I answered your question or solved your problem, mark this post as the solution.

[tayloramy]

Hi @tan_thiamhuat, 

No, environments are specific to the workspace they are in, you can't share an environment between workspaces. 

Create, Configure, and Use an Environment in Fabric - Microsoft Fabric | Microsoft Learn

If you found this helpful, consider giving some Kudos. If I answered your question or solved your problem, mark this post as the solution.

[tan_thiamhuat · Accepted solution]

this still has not solved my issue. Everything I need those identical libraries to be used in other workspaces, I need to create another environment in the workspace, and the creation of new environment takes a long time. Can Microsoft looks into how to have a better solution to this?

[tan_thiamhuat]

yes, I understand that.. but I tested out. At time, it works across different workspaces. That is why I get confused.

[tayloramy]

Hi @tan_thiamhuat, 

Sorry, I stand corrected. Seems the MVPs from the Reactor session today were also incorrect: 
Create, Configure, and Use an Environment in Fabric - Microsoft Fabric | Microsoft Learn

---

The environment is available on both the Notebook and Spark Job Definition tabs. When notebooks and Spark job definitions are attached to an environment, they can access its libraries, compute configurations, and resources. The Explorer lists all available environments that are shared with you, are from the current workspace, and are from other workspaces to which you have access.

When you attach an environment from another workspace, both workspaces must have the same capacity and network security settings. Although you can select environments from workspaces with different capacities or network security settings, the session fails to start.

When you attach an environment from another workspace, the compute configuration in that environment is ignored. Instead, the pool and compute configurations default to the settings of your current workspace.

---

It seems this is supported. 

Are your workspaces on the same capacity and have the same network security settings? 

If you found this helpful, consider giving some Kudos. If I answered your question or solved your problem, mark this post as the solution.

[tan_thiamhuat]

I am using the free tier (I suppose no capacity, so it is the same capacity), and same network security settings. How can we create an environment outside of any workspace, so that whichever PySpark notebook in whichever workspace can access that environment without any issue? Else users would need to duplicate the same environment in each workspace if their PySpark notebooks in different workspaces have identical needs for that environment. Not sure if my requirements are clear.

[tayloramy]

Hi @tan_thiamhuat, 

Do you mean you're on a free trial capacity? There is no free tier of Fabric, and to be able to create fabric items like notebooks and environments, the workspace needs to be attached to a fabric capcaity. 

Fabric items can't exist outside of workspaces, so you can't make an environment outside of any workspace. 

If you found this helpful, consider giving some Kudos. If I answered your question or solved your problem, mark this post as the solution. 

[tan_thiamhuat]

there is a Trial period at the top of my Fabric.

[tayloramy]

Are all workspaces that you're trying to use the encironment in attached to the same trial capacity? You can check this in the wprkspace settings, in the license tab. 

If you found this helpful, consider giving some Kudos. If I answered your question or solved your problem, mark this post as the solution.