Forum Discussion

reddyr2502's avatar
reddyr2502
New Member
9 days ago
Solved

Fabric Workspace Architecture

I'm a Data Analytics Engineer at a logistics startup, and I'm the only person handling the data space. I'm looking for insights on how to effectively manage and monitor ETL processes using metadata-driven pipelines for a medallion architecture.

 

I have 10 applications with about 2k tables total, though NetSuite accounts for 40% of the data. I'm considering 3 approaches for the Fabric workspace architecture:

 

Approach 1:

Create a dedicated NetSuite workspace with bronze, silver, and gold pipelines, plus 3 lakehouses for the medallion layers and schemas for individual departments. Then create similar workspaces for other applications.

 

Approach 2:

Organize by layer instead—separate workspaces for raw-to-bronze, bronze-to-silver, and silver-to-gold, with each workspace handling all applications at that layer.

 

Approach 3:

Use a single workspace to manage all medallion layers for all applications, but I'm not sure how to easily manage and monitor all the pipelines in this setup.

 

Can you give me the pros and cons for each approach and recommend which one is best for a solo data engineer?

  • As the use case has limited 1 Data Engineer, I would recommend it to go with Option 3 as it will have :

    • Very easy to manage
    • Everything in one place
    • Easier monitoring
    • Less clicking around and
    • Faster development.

    But i will also have few drawbacks like, can become messy if naming standards are poor, less security control etc.

    Once the project mature and team expand you can gradually shift towards option-2 or Option-1

9 Replies

  • Hi reddyr2502

     

    This depends on your requirements. Is there a need for data seperation between layers (either for security reasons, or company policy?) 

    It also depends on how many items you have in the workspace, there is a limit to the number of items in a single workspace, though my opinion is that if you hit that limit you've done something wrong. 

     

    In my enviornment, I have all three layers in a single workspace. My bronze and silver layers are fully automated and controlled through metadata, where my gold layer is bespoke for each final table. This sounds similar to what you are trying to do, though I am doing it with notebooks and not with pipelines. 



    I would advise against creating a new workspace just for NetSuite as that will just create data silos and lead to challenges later when you want to integrate data from multiple sources into a single report. 

    Approach 2 can make sense if you need seperation between the layers, or if your design has hundreds of items per layer. 

    Approach 3 is the simpliest, and that's my vote. 

  • reddyr2502​ 
    The best approach is use meddalion architecture in single workspace for Dev and then deploy the same architecture in UAT workspace and then PROD workspace.
    you need to create three lakehouses. 

    lh_bronze ---> lh_silver ---> lh_gold

     If this helps, ✓ Mark as Kudos | Help Others

  • Kagiyama_yutaka's avatar
    Kagiyama_yutaka
    Responsive Resident

    Single workspace is the simplest for a solo engineer, and Fabric keeps bronze, silver, and gold in one place without cross‑workspace hops. This gives a single view for lineage, failures, and dependencies. Group pipelines by app and layer so the medallion flow stays clear without the overhead of multi‑workspace setups.

  • ShivekMaharaj's avatar
    ShivekMaharaj
    Impactful Individual

    Hi reddyr2502,

     

    I would lean toward Approach 3 as well, especially for a solo engineer, but I would base the workspace boundaries more on security, ownership and lifecycle requirements than on the individual source systems.

     

    Microsoft's current Fabric deployment guidance explicitly supports a single-workspace medallion design with separate lakehouses for bronze, silver and gold. For a smaller team, that keeps orchestration and day-to-day management much simpler.

     

    I also would not give NetSuite its own workspace purely because it represents 40% of the data. If the workspaces sit on the same Fabric capacity, they still share that capacity, so separating them doesn't inherently provide compute isolation.

     

    If you later need a stronger security or ownership boundary, a useful evolution is bronze + silver in a staging workspace and gold + semantic models/reports in a presentation workspace. Microsoft actually uses that pattern in its Fabric CI/CD guidance.

     

    One other small point: I wouldn't use the 2,000-table count alone as a reason to split. Fabric does have a 1,000-item workspace limit, but that refers to Fabric/Power BI workspace items. A lakehouse itself is an item and can contain many tables.

     

    So for where you are now, I would start with Approach 3 and introduce additional workspaces when an actual security, ownership or deployment boundary appears.

  • ipkus's avatar
    ipkus
    Frequent Visitor

    Approach 3 might be the easiest to manage long term. From what I understand, you most likely have a single capacity. You could create the required Lakehouses and Warehouses within one workspace and use security to provide the appropriate level of access to each team.

     

    If each department’s data needs to be separated, using a separate schema for each department could be a good approach. However, we would need a little more detail about the requirements to recommend the right structure.

     

    In most cases, I would not recommend giving users access to the entire workspace. Instead, provide access only to the specific Lakehouse or Warehouse they need.

  • jainpraful's avatar
    jainpraful
    Regular Visitor

    Hi reddyr2502​,

    Honestly, since it's just you handling ~2k tables across 10 apps, your biggest enemy is admin overhead.
    Here’s how I’d look at the three setups:

    Per Application: It sounds nice for isolating compute, but it will quickly turn into a headache. You'll end up maintaining 10+ workspaces solo, and building unified models in Gold (like joining NetSuite finance data with your logistics data) gets messy with tons of cross-workspace sharing.

    Single Workspace: Super easy to start, but with that many tables, notebooks, and semantic models, it will turn into a junk drawer real fast. Plus, workspace-level permissions make it tough to keep users out of staging tables.

    Per Medallion Layer: This is usually the sweet spot. Having dedicated workspaces for Bronze, Silver, and Gold keeps things clean, gives you natural access boundaries, and makes joining cross-app data in Gold way simpler.

    My recommendation: Go with the layer-based approach (Approach 2). 


    1 - Ditch 1:1 pipelines: With 2k tables, definitely don't build pipelines manually. Build a metadata table (source table, watermarks, refresh cadence) and run a couple of generic, parameterized pipelines/notebooks off of it.

    2 - Organize Bronze by schema: Keep your raw tables organized inside your Bronze lakehouse using prefixes or schemas like netsuite_orders or wms_inventory so nothing gets lost.

    3 - Monitoring: The built-in Fabric Monitoring Hub is great for checking failed runs. If you want quick visual health checks, just have your pipelines append execution logs (row counts, duration, status) into a simple telemetry Delta table and throw a basic Power BI page over it.

     

    Hope that helps point you in the right direction!

  • As the use case has limited 1 Data Engineer, I would recommend it to go with Option 3 as it will have :

    • Very easy to manage
    • Everything in one place
    • Easier monitoring
    • Less clicking around and
    • Faster development.

    But i will also have few drawbacks like, can become messy if naming standards are poor, less security control etc.

    Once the project mature and team expand you can gradually shift towards option-2 or Option-1