Forum Discussion
Fabric Workspace Architecture
As the use case has limited 1 Data Engineer, I would recommend it to go with Option 3 as it will have :
- Very easy to manage
- Everything in one place
- Easier monitoring
- Less clicking around and
- Faster development.
But i will also have few drawbacks like, can become messy if naming standards are poor, less security control etc.
Once the project mature and team expand you can gradually shift towards option-2 or Option-1
Hi reddyr2502,
Honestly, since it's just you handling ~2k tables across 10 apps, your biggest enemy is admin overhead.
Here’s how I’d look at the three setups:
Per Application: It sounds nice for isolating compute, but it will quickly turn into a headache. You'll end up maintaining 10+ workspaces solo, and building unified models in Gold (like joining NetSuite finance data with your logistics data) gets messy with tons of cross-workspace sharing.
Single Workspace: Super easy to start, but with that many tables, notebooks, and semantic models, it will turn into a junk drawer real fast. Plus, workspace-level permissions make it tough to keep users out of staging tables.
Per Medallion Layer: This is usually the sweet spot. Having dedicated workspaces for Bronze, Silver, and Gold keeps things clean, gives you natural access boundaries, and makes joining cross-app data in Gold way simpler.
My recommendation: Go with the layer-based approach (Approach 2).
1 - Ditch 1:1 pipelines: With 2k tables, definitely don't build pipelines manually. Build a metadata table (source table, watermarks, refresh cadence) and run a couple of generic, parameterized pipelines/notebooks off of it.
2 - Organize Bronze by schema: Keep your raw tables organized inside your Bronze lakehouse using prefixes or schemas like netsuite_orders or wms_inventory so nothing gets lost.
3 - Monitoring: The built-in Fabric Monitoring Hub is great for checking failed runs. If you want quick visual health checks, just have your pipelines append execution logs (row counts, duration, status) into a simple telemetry Delta table and throw a basic Power BI page over it.
Hope that helps point you in the right direction!