Forum Discussion

icassiem's avatar
icassiem
Post Prodigy
1 month ago
Solved

Medallion Archtecture

Good day, Kindly advise on what the modern archietecture for a fabric medallion of dilver and gold only, currently reading API & possibly semi structured data too so my research says silver is my l...
data engineering
  • Parchitect's avatar
    Parchitect
    1 month ago
    Hi icassiem,

     

    No silly questions at all. These are exactly the right questions when you start thinking about data architecture instead of only solving one report.

     

    The goal is to avoid a spaghetti architecture later.
    I would separate two things:
    1. What you know today
    2. What might happen later

     

    Do not over-design only based on assumptions. But also do not name everything so narrowly that it becomes hard to grow.

     

    For your case, I would choose a balanced approach:
    Workspace:
    ws-analytics-dev
    ws-analytics-prod

     

    Then keep the domain in the item/table names:
    lh_silver
    wh_gold

     

    Tables or schemas:
    silver_product_customer
    silver_product_sales
    gold_dim_customer
    gold_fact_sales

     

    If Product becomes big enough later, or IT/Finance needs different ownership/security/lifecycle, then split into domain workspaces later:
    ws-product-analytics-prod
    ws-it-analytics-prod

     

    So yes, start with what is factual today, but use naming that does not block future growth.

     

    For the medallion question:
    Yes, it is still medallion even if the physical items are different.
    Medallion is a logical pattern, not a rule that every layer must be the same technology.

     

    Example:
    Silver = Lakehouse
    Gold = Warehouse
    Optional forecasting output = Gold Lakehouse or ML output table

     

    That is still a medallion architecture because the logic is:
    Source → cleaned/standardized → curated/business-ready

     

    I would not create lh_gold unless you have a clear need for it.
    For now, I would keep it simple:

     

    API / JSON / CSV
    → lh_silver
    → wh_gold
    → semantic model
    → Power BI / Copilot narratives

     

    Add lh_gold later only if forecasting/Python needs curated Delta output before loading to the Warehouse.

     

    Architecture criteria should be based on:
    - ownership
    - security
    - lifecycle
    - performance
    - data reuse
    - maintainability

     

    Not only “possible future request”.
Parchitect's avatar
Parchitect
Solution Sage
1 month ago
Hi icassiem,

 

I would keep this post focused on the medallion design. For pipeline patching/maintenance, I suggest opening a separate post so this thread does not become too broad.

 

For your setup on F2, I would keep it simple:
API / JSON / CSV
→ Silver Lakehouse
→ Gold Warehouse
→ Semantic model
→ Power BI reports / Copilot narratives

 

1. Workspace naming
If you are not sure whether this will stay Product-only, I would avoid naming the workspace too narrowly.

 

Use something broader like:
ws-analytics-dev
ws-analytics-prod
or:
ws-commercial-analytics-dev
ws-commercial-analytics-prod

 

Then keep the domain in table/item names, for example:
silver_product_customer
silver_product_sales
gold_dim_customer
gold_fact_sales

 

If Product or IT grows later, then split into separate domain workspaces.

 

2. Gold Lakehouse + Gold Warehouse?
You do not need both from day one.

 

Start with:
Silver Lakehouse = API, JSON, semi-structured data, Python/Spark transformations 
Gold Warehouse = SQL marts, dim/fact tables, reporting-ready layer
Only add a Gold Lakehouse later if you need Python/forecasting outputs stored as curated Delta tables.

 

3. JSON / semi-structured data
For simple JSON, Dataflow Gen2 may be enough.
For complex JSON, nested arrays, API pagination, or schema drift, use a Fabric Notebook with Python/PySpark and write the result to the Silver Lakehouse.

 

4. Domain split
Do not create too many Lakehouses at the start.

 

Start simple:
lh_silver
wh_gold

 

Split by domain later only if you need different ownership, security, lifecycle, or performance boundaries.

 

Short recommendation:
- Silver = Lakehouse
- Gold reporting marts = Warehouse
- Forecasting/Python output = optional Gold Lakehouse later
- Copilot narratives depend more on the semantic model/report design than whether Gold is Lakehouse or Warehouse
icassiem's avatar
icassiem
Post Prodigy
1 month ago

Parchitect Thank You

1. Workspace = "Analytics"

2. Lkh Silver + Whs Gold = "Domain" Product / IT etc

3. Add Lhs Gold when FC needs, is it then still a medallion if lh_silver, wh_gold + lh_gold = 3 layers actually 4 if silver adds?

4. Does this mean "If Product or IT grows later, then split into separate domain workspaces."  that under ws "analytics" i have lh_silver_Domain or should i only build what the need is now meaning i know its product iam guessing it could eveolve assumption, should i just focus on what is fact, like WS Product Analytics, lh_gol_tbl? How are arch criteria defined i guess what im lpoking for fact or possible assumption but its uneccessary broad or just push back when the need arrives as not part of design but then its not enterprise?

 

Apologies for the silly questions