Forum Discussion
Data Pipelines for cross-workspace medallion execution
I just got back from FabCon in Atlanta, and the expert at the MS Data Engineering table told me that I should have separate workspaces for bronze, silver, and gold, as this helps with capacity and security segmentation. It is assumed that these separate workspaces will contain one or more Lakehouses each. I plan to include Notebooks, and Variable Libraries in these workspaces respectively. My question is how would YOU structure Data Pipelines for this? Should I have a data pipeline in each workspace, unscheduled, and some "master" data pipeline elsewhere that orchestrates the execution of each bronze, silver, and gold data pipeline sequentially? Where should said "master" data pipeline live, if so? I'm open to suggestions as I'm treating this as a whole new architectural design for when we move to a new, clean-slate Fabric tenant.
Hi seanrich ,
Yes — use the "master orchestrator" pattern.
Create a fourth workspace called Orchestration or DataOps. This is your control plane.
Each medallion workspace (Bronze, Silver, Gold) keeps its own Lakehouses, Notebooks, Variable Libraries, and one or more unscheduled worker pipelines (e.g., Bronze_Ingest_Main). These never run on their own — they only fire when called.
The Orchestration workspace holds your master pipeline, which uses the Invoke Pipeline activity to call each worker sequentially: Bronze → Silver → Gold. It also holds your schedule triggers, failure alerting logic, and optionally a small Lakehouse for audit logging.
Why not put the master in one of the medallion workspaces? Three reasons: capacity isolation (a heavy Bronze ingestion won't starve your orchestrator), security segmentation (not everyone who edits Bronze notebooks should control the master schedule), and clean separation of concerns (Bronze shouldn't implicitly "own" the whole chain).
Since you're on a clean-slate tenant, also plan your environment strategy early — decide whether you'll have Dev_Bronze, Dev_Silver, etc. or use deployment pipelines to promote, because that multiplies your workspace count fast.
This pattern scales well and keeps the mental model obvious for anyone joining the team.
If this post helps, then please appreciate giving a Kudos or accepting as a Solution to help the other members find it more quickly.
If I misunderstand your needs or you still have problems on it, please feel free to let me know. Thanks a lot!Hi seanrich ,
One orchestration workspace per source system (orch-salesforce, orch-replicon, etc.). Each one owns the sequential bronze → silver → gold chain for its source. They run independently, potentially on different schedules — maybe Salesforce is near-real-time while Replicon is a nightly batch. This keeps the blast radius small: a Replicon failure doesn't block Salesforce processing, and whoever owns a source system manages its own cadence.
Then one Platform Orchestration workspace sits above everything. This is your thin coordination layer — it doesn't micromanage notebook execution. Its responsibilities are:
• Trigger domain-layer refreshes (ticketing-shortcuts, sales-opps-shortcuts, etc.) once the relevant upstream sources have landed
• Trigger audience workspace refreshes (finance, sales, customer-support) after their dependent domains are complete
• Centralized alerting, audit logging, and health dashboardsWhy not one giant orchestrator? At this scale you'd have dozens of Invoke Pipeline activities, different sources needing different schedules, and one failure potentially blocking unrelated downstream work. Why not an orchestrator per workspace? Too granular — you'd lose the ability to express "after all sales-related sources land, refresh the sales-opps domain."
On the environment strategy — since you mentioned UAT and prod via additional workspaces, I'd strongly recommend using Fabric deployment pipelines to promote content rather than manually maintaining separate dev/uat/prod copies of every workspace. That multiplies your workspace count fast and creates drift risk.
So your prod workspace inventory would look roughly like:
• Source orchestration: orch-salesforce, orch-replicon, etc.
• Source medallion: bronze/silver/gold per source (as you have)
• Domain: as you've outlined
• Audience: as you've outlined
• Platform orchestration: one workspace at the topClean, independently securable, capacity-isolated, and easy for someone new to reason about.
If this post helps, then please appreciate giving a Kudos or accepting as a Solution to help the other members find it more quickly.
If I misunderstand your needs or you still have problems on it, please feel free to let me know. Thanks a lot!
4 Replies
Hi seanrich ,
Yes — use the "master orchestrator" pattern.
Create a fourth workspace called Orchestration or DataOps. This is your control plane.
Each medallion workspace (Bronze, Silver, Gold) keeps its own Lakehouses, Notebooks, Variable Libraries, and one or more unscheduled worker pipelines (e.g., Bronze_Ingest_Main). These never run on their own — they only fire when called.
The Orchestration workspace holds your master pipeline, which uses the Invoke Pipeline activity to call each worker sequentially: Bronze → Silver → Gold. It also holds your schedule triggers, failure alerting logic, and optionally a small Lakehouse for audit logging.
Why not put the master in one of the medallion workspaces? Three reasons: capacity isolation (a heavy Bronze ingestion won't starve your orchestrator), security segmentation (not everyone who edits Bronze notebooks should control the master schedule), and clean separation of concerns (Bronze shouldn't implicitly "own" the whole chain).
Since you're on a clean-slate tenant, also plan your environment strategy early — decide whether you'll have Dev_Bronze, Dev_Silver, etc. or use deployment pipelines to promote, because that multiplies your workspace count fast.
This pattern scales well and keeps the mental model obvious for anyone joining the team.
If this post helps, then please appreciate giving a Kudos or accepting as a Solution to help the other members find it more quickly.
If I misunderstand your needs or you still have problems on it, please feel free to let me know. Thanks a lot!Thank you for the detail! Would this Orchestration workspace service many other workspaces, or should it just service the one set of bronze, silver, gold? For further context, the architecture would look like the following. Each source-system item below is a workspace, and will be persisted to uat and prod via additional workspaces. Same with domain, and audience layers.
source_system
-salesforce-bronze
-salesforce-silver
-salesforce-gold
-replicon-bronze
-etc
domain
-ticketing-shortcuts
-ticketing-gold
-sales-opps-shortcuts
-sales-opps-gold
audience
-finance
-sales
-customer-support
Hi seanrich ,
Great follow-up — and with that architecture, a single monolithic orchestrator would get unwieldy fast. I'd recommend a hybrid approach. This widely resonates to what they demonstrated in the workshop as well.
One orchestration workspace per source system, plus one platform-level orchestrator on top.
Source-level orchestrators (e.g., orch-salesforce, orch-replicon) each own their bronze → silver → gold chain for that source. They run independently, potentially on different schedules — Salesforce might be near-real-time while Replicon is a nightly batch. This keeps the blast radius small: a Replicon failure doesn't block Salesforce, and each source team manages its own cadence.
Then a single Platform Orchestration workspace sits above those. Its job is cross-cutting coordination:
- Trigger domain-layer refreshes (ticketing-shortcuts, sales-opps-shortcuts, etc.) once the relevant upstream sources have landed
- Trigger audience workspace refreshes (finance, sales, customer-support) after the domains they depend on are complete
- Centralized alerting, audit logging, and health monitoringWhy not one giant orchestrator calling everything? At your scale you'd have dozens of Invoke Pipeline activities in a single pipeline, different sources needing different schedules, and one failure potentially blocking unrelated downstream work. Why not an orchestrator per workspace? Too granular — you'd lose the ability to express "after all sales-related sources land, refresh the sales domain."
For the environment strategy (dev/UAT/prod), I'd strongly recommend using Fabric deployment pipelines to promote content across environments rather than creating dev-salesforce-bronze, uat-salesforce-bronze, prod-salesforce-bronze, etc. That triples your workspace count immediately. Keep the same workspace topology in each environment and promote artifacts through stages. The orchestration workspaces get promoted the same way.
So your prod workspace inventory would look roughly like:
- Source orchestration: orch-salesforce, orch-replicon, etc. (one per source)
- Source medallion: bronze/silver/gold per source (as you have)
- Domain: as you've outlined
- Audience: as you've outlined
- Platform orchestration: one workspace at the topThis keeps everything independently securable, capacity-isolated, and easy to reason about when someone new joins the team.
If this post helps, then please appreciate giving a Kudos or accepting as a Solution to help the other members find it more quickly.
If I misunderstand your needs or you still have problems on it, please feel free to let me know. Thanks a lot!Hi seanrich ,
One orchestration workspace per source system (orch-salesforce, orch-replicon, etc.). Each one owns the sequential bronze → silver → gold chain for its source. They run independently, potentially on different schedules — maybe Salesforce is near-real-time while Replicon is a nightly batch. This keeps the blast radius small: a Replicon failure doesn't block Salesforce processing, and whoever owns a source system manages its own cadence.
Then one Platform Orchestration workspace sits above everything. This is your thin coordination layer — it doesn't micromanage notebook execution. Its responsibilities are:
• Trigger domain-layer refreshes (ticketing-shortcuts, sales-opps-shortcuts, etc.) once the relevant upstream sources have landed
• Trigger audience workspace refreshes (finance, sales, customer-support) after their dependent domains are complete
• Centralized alerting, audit logging, and health dashboardsWhy not one giant orchestrator? At this scale you'd have dozens of Invoke Pipeline activities, different sources needing different schedules, and one failure potentially blocking unrelated downstream work. Why not an orchestrator per workspace? Too granular — you'd lose the ability to express "after all sales-related sources land, refresh the sales-opps domain."
On the environment strategy — since you mentioned UAT and prod via additional workspaces, I'd strongly recommend using Fabric deployment pipelines to promote content rather than manually maintaining separate dev/uat/prod copies of every workspace. That multiplies your workspace count fast and creates drift risk.
So your prod workspace inventory would look roughly like:
• Source orchestration: orch-salesforce, orch-replicon, etc.
• Source medallion: bronze/silver/gold per source (as you have)
• Domain: as you've outlined
• Audience: as you've outlined
• Platform orchestration: one workspace at the topClean, independently securable, capacity-isolated, and easy for someone new to reason about.
If this post helps, then please appreciate giving a Kudos or accepting as a Solution to help the other members find it more quickly.
If I misunderstand your needs or you still have problems on it, please feel free to let me know. Thanks a lot!
