Environment for Python packages to be used by different PySpark scripts in different Workspaces
let us say I have created a new Environment named Environment_A for all the different Python packages I need for the PySpark script to run inside Workspace_A. Can this Environment_A be used for another PySpark script running in Workspace_B? I find at times, it works, and other times, it gets disconnected instantly.
The question then is how can I create a new Environment to be used in different workspaces for different PySpark scripts? You know create a new environment with the different Python packages takes time, and we do not want to duplicate the same environment all over the different workspaces.
What is the best solution then?
Everything I need those identical libraries to be used in other workspaces, I need to create another environment in the workspace, and the creation of new environment takes a long time. Can Microsoft looks into how to have a better solution to this?
Recent ideas
Data Pipelines - Run only selected activities
For debugging and testing pipeline activities during development, allow us to select one or multiple activities and run only the selected pipeline activities. For example, I'm working on editing ...frithjof_v21 hours agoCommunity ChampionNew618Views11likes2CommentsSemantic model connection bindings should be in source control (Git)
Semantic model data source connection bindings should be source controlled. A semantic model can contain multiple data source references, each of which can be mapped to a separate Fabric data connec...frithjof_v1 day agoCommunity ChampionNew21Views1like0CommentsBulk changing column names in Visualizations Pane
We often use raw/api column names or measures with a set nomenclature to be consistent and to keep track of them but we do not want to display these names in the visuals. Currently we have to change ...vishal1401971 day agoFrequent VisitorNew6Views0likes0Comments