Forum Discussion

alexandremichi's avatar
alexandremichi
Regular Visitor
1 day ago
Solved

Sharing tooling across Fabric capacities

Hi all, Following the recommended multi-workspace, multi-capacity pattern, we use separate workspaces for different ETL activities. Shared tooling and Environments live in a central workspace, which...
  • jubinsoni's avatar
    1 day ago

    Hi alexandremichi​,

    Echoing Murtaza_Ghafoor​: as far as I know, an Environment can only be attached to items on the same capacity, so the dependable pattern is one Environment per capacity, kept identical through source control and automation instead of by hand. I have not seen a published date for cross-capacity sharing, so the Fabric roadmap (release plan) is the place to confirm it. Below is how I would make "one per capacity" cheap to maintain.

    1. Treat the Environment as code

    The point is that the central tooling workspace stops being the source of truth, and a Git repository becomes it. Each capacity gets a copy of the same definition:

    Repository
      environments/
        etl-tooling/
          Libraries/         -> custom .whl files, public/custom library lists
          Setting/           -> Spark compute and runtime settings
          environment.yml    -> library definition
      notebooks/ pipelines/  -> items that reference the environment by name

    2. Deploy the same definition to every capacity

    Pick one promotion mechanism and use it for the Environment as well as for the other items, so Dev and Prod can never drift:

    Option A  Fabric Git integration + deployment pipelines -> promote Dev -> Test -> Prod stages
    Option B  fabric-cicd (Python) from Azure DevOps       -> publish items per target workspace
    Option C  REST API / Fabric CLI                         -> publish and update the environment from a pipeline step

    A sketch of option B, with the environment type included in the publish and the target workspace chosen per stage:

    from fabric_cicd import FabricWorkspace, publish_all_items
    
    target = FabricWorkspace(
        workspace_id="<dev-or-prod-workspace-id>",
        environment="DEV",                       # parameter.yml selects stage values
        repository_directory="./",
        item_type_in_scope=["Environment", "Notebook", "DataPipeline"],
    )
    publish_all_items(target)

    3. Keep what differs by stage out of the definition

    Same in every stage  -> libraries, Spark settings, runtime version
    Different per stage  -> connection details, lakehouse names, workspace IDs (Variable Library or parameter file)
    Versioning           -> pin library versions and tag the repo so Dev and Prod can be compared quickly

    4. Options if you want to avoid duplicates entirely

    • Publish shared Python code as a versioned wheel in one artifact feed or storage location and have each Environment reference the same wheel version, so there is one build and many consumers.
    • For small helper code, inline installs in the notebook (with pinned versions) avoid an Environment at the cost of slower session startup.
    • If sharing across workspaces is the real goal rather than sharing across capacities, keeping dev and prod on one capacity is the simplest option, at the price of the isolation you gain from separate capacities.

    So my overall approach would be:

    Source of truth -> Git repo with the Environment definition
    Deploy          -> same definition published to each capacity by CI/CD
    Stage values    -> Variable Library / parameters, never baked into the Environment
    Roadmap         -> confirm cross-capacity Environment sharing in the release plan

    AI-assisted drafting: AI was used to help structure and phrase this response. I reviewed and validated the technical content before posting.