Forum Discussion

icassiem's avatar
icassiem
Post Prodigy
1 month ago
Solved

Medallion Archtecture

Good day, Kindly advise on what the modern archietecture for a fabric medallion of dilver and gold only, currently reading API & possibly semi structured data too so my research says silver is my l...
data engineering
  • Parchitect's avatar
    Parchitect
    1 month ago
    Hi icassiem,

     

    No silly questions at all. These are exactly the right questions when you start thinking about data architecture instead of only solving one report.

     

    The goal is to avoid a spaghetti architecture later.
    I would separate two things:
    1. What you know today
    2. What might happen later

     

    Do not over-design only based on assumptions. But also do not name everything so narrowly that it becomes hard to grow.

     

    For your case, I would choose a balanced approach:
    Workspace:
    ws-analytics-dev
    ws-analytics-prod

     

    Then keep the domain in the item/table names:
    lh_silver
    wh_gold

     

    Tables or schemas:
    silver_product_customer
    silver_product_sales
    gold_dim_customer
    gold_fact_sales

     

    If Product becomes big enough later, or IT/Finance needs different ownership/security/lifecycle, then split into domain workspaces later:
    ws-product-analytics-prod
    ws-it-analytics-prod

     

    So yes, start with what is factual today, but use naming that does not block future growth.

     

    For the medallion question:
    Yes, it is still medallion even if the physical items are different.
    Medallion is a logical pattern, not a rule that every layer must be the same technology.

     

    Example:
    Silver = Lakehouse
    Gold = Warehouse
    Optional forecasting output = Gold Lakehouse or ML output table

     

    That is still a medallion architecture because the logic is:
    Source → cleaned/standardized → curated/business-ready

     

    I would not create lh_gold unless you have a clear need for it.
    For now, I would keep it simple:

     

    API / JSON / CSV
    → lh_silver
    → wh_gold
    → semantic model
    → Power BI / Copilot narratives

     

    Add lh_gold later only if forecasting/Python needs curated Delta output before loading to the Warehouse.

     

    Architecture criteria should be based on:
    - ownership
    - security
    - lifecycle
    - performance
    - data reuse
    - maintainability

     

    Not only “possible future request”.
icassiem's avatar
icassiem
Post Prodigy
1 month ago

Parchitect Thank You

1. Workspace = "Analytics"

2. Lkh Silver + Whs Gold = "Domain" Product / IT etc

3. Add Lhs Gold when FC needs, is it then still a medallion if lh_silver, wh_gold + lh_gold = 3 layers actually 4 if silver adds?

4. Does this mean "If Product or IT grows later, then split into separate domain workspaces."  that under ws "analytics" i have lh_silver_Domain or should i only build what the need is now meaning i know its product iam guessing it could eveolve assumption, should i just focus on what is fact, like WS Product Analytics, lh_gol_tbl? How are arch criteria defined i guess what im lpoking for fact or possible assumption but its uneccessary broad or just push back when the need arrives as not part of design but then its not enterprise?

 

Apologies for the silly questions

 

Parchitect's avatar
Parchitect
Solution Sage
1 month ago
Hi icassiem,

 

No silly questions at all. These are exactly the right questions when you start thinking about data architecture instead of only solving one report.

 

The goal is to avoid a spaghetti architecture later.
I would separate two things:
1. What you know today
2. What might happen later

 

Do not over-design only based on assumptions. But also do not name everything so narrowly that it becomes hard to grow.

 

For your case, I would choose a balanced approach:
Workspace:
ws-analytics-dev
ws-analytics-prod

 

Then keep the domain in the item/table names:
lh_silver
wh_gold

 

Tables or schemas:
silver_product_customer
silver_product_sales
gold_dim_customer
gold_fact_sales

 

If Product becomes big enough later, or IT/Finance needs different ownership/security/lifecycle, then split into domain workspaces later:
ws-product-analytics-prod
ws-it-analytics-prod

 

So yes, start with what is factual today, but use naming that does not block future growth.

 

For the medallion question:
Yes, it is still medallion even if the physical items are different.
Medallion is a logical pattern, not a rule that every layer must be the same technology.

 

Example:
Silver = Lakehouse
Gold = Warehouse
Optional forecasting output = Gold Lakehouse or ML output table

 

That is still a medallion architecture because the logic is:
Source → cleaned/standardized → curated/business-ready

 

I would not create lh_gold unless you have a clear need for it.
For now, I would keep it simple:

 

API / JSON / CSV
→ lh_silver
→ wh_gold
→ semantic model
→ Power BI / Copilot narratives

 

Add lh_gold later only if forecasting/Python needs curated Delta output before loading to the Warehouse.

 

Architecture criteria should be based on:
- ownership
- security
- lifecycle
- performance
- data reuse
- maintainability

 

Not only “possible future request”.
  • icassiem's avatar
    icassiem
    Post Prodigy
    1 month ago

    Parchitect  wow thank you 😊

     

    does this "ws-product-analytics-prod + ws-it-analytics-prod" mean 2 workspaces and F2 license wont work as i think it only allows for one workspace and then i will have different silv+gold arch correct? probably best just to keep it universal the 1 workspace like "silver_product_sales & silver_IT_storage"

    • Parchitect's avatar
      Parchitect
      Solution Sage
      1 month ago
      No, F2 does not mean you can only have one workspace. You can have multiple workspaces assigned to the same Fabric capacity.

       

      The important point is that they all share the same F2 compute. So technically you can have:

       ws-product-analytics-prod

      ws-it-analytics-prod

       

      on the same F2 capacity, but on a small F2 SKU I would only split workspaces when there is a real reason. 
      Based on your situation, I agree with your thinking: keep it simple and use one broader workspace.