Forum Discussion

icassiem's avatar
icassiem
Post Prodigy
1 month ago
Solved

Medallion Archtecture

Good day, Kindly advise on what the modern archietecture for a fabric medallion of dilver and gold only, currently reading API & possibly semi structured data too so my research says silver is my l...
data engineering
  • Parchitect's avatar
    Parchitect
    1 month ago
    Hi icassiem,

     

    No silly questions at all. These are exactly the right questions when you start thinking about data architecture instead of only solving one report.

     

    The goal is to avoid a spaghetti architecture later.
    I would separate two things:
    1. What you know today
    2. What might happen later

     

    Do not over-design only based on assumptions. But also do not name everything so narrowly that it becomes hard to grow.

     

    For your case, I would choose a balanced approach:
    Workspace:
    ws-analytics-dev
    ws-analytics-prod

     

    Then keep the domain in the item/table names:
    lh_silver
    wh_gold

     

    Tables or schemas:
    silver_product_customer
    silver_product_sales
    gold_dim_customer
    gold_fact_sales

     

    If Product becomes big enough later, or IT/Finance needs different ownership/security/lifecycle, then split into domain workspaces later:
    ws-product-analytics-prod
    ws-it-analytics-prod

     

    So yes, start with what is factual today, but use naming that does not block future growth.

     

    For the medallion question:
    Yes, it is still medallion even if the physical items are different.
    Medallion is a logical pattern, not a rule that every layer must be the same technology.

     

    Example:
    Silver = Lakehouse
    Gold = Warehouse
    Optional forecasting output = Gold Lakehouse or ML output table

     

    That is still a medallion architecture because the logic is:
    Source → cleaned/standardized → curated/business-ready

     

    I would not create lh_gold unless you have a clear need for it.
    For now, I would keep it simple:

     

    API / JSON / CSV
    → lh_silver
    → wh_gold
    → semantic model
    → Power BI / Copilot narratives

     

    Add lh_gold later only if forecasting/Python needs curated Delta output before loading to the Warehouse.

     

    Architecture criteria should be based on:
    - ownership
    - security
    - lifecycle
    - performance
    - data reuse
    - maintainability

     

    Not only “possible future request”.
Parchitect's avatar
Parchitect
Solution Sage
1 month ago
Hi  , 
So short answer:
- Yes, you can move Gold marts to Fabric Warehouse.
- But keep forecasting / Python-friendly curated tables in Gold Lakehouse if needed.
- For reporting, SQL marts, dimensional modelling, and BI consumption, Gold Warehouse is a good choice.
- For F2, keep it simple and avoid creating too many separate workspaces/items too early.

 

Longer answer:
For your scenario, I would not think of this as “Lakehouse OR Warehouse”. In Fabric it is very common to use both, depending on the purpose of the layer.

For your case:
- Silver = Lakehouse
- Gold = Lakehouse and/or Warehouse depending on consumption pattern

 

Since you are ingesting APIs and possibly semi-structured data, Silver in a Lakehouse is the right place. Lakehouse is better suited for Spark/Python, semi-structured data, data engineering, and future ML/forecasting workloads. Microsoft’s decision guide also points Lakehouse toward Spark, mixed/semi-structured data, big data, and machine learning scenarios.

 

For Gold, I would use this rule:
Use Gold Lakehouse when:
- you still need Spark/Python transformations
- you want to keep Delta tables for data science / forecasting
- the data is still evolving
- you want to train or score models with notebooks

 

Use Fabric Warehouse when:
- the data is structured and business-ready
- you want SQL-based marts
- you want star schemas / dimensional models
- your consumers are BI users and SQL users
- you need stronger SQL modelling patterns and curated serving tables

 

So a good pattern would be:
API / semi-structured source
→ Silver Lakehouse: cleaned, standardized, deduplicated Delta tables
→ Gold Lakehouse: curated analytical Delta tables and forecasting outputs
→ Gold Warehouse: SQL marts / dimensional model for reporting and business consumption
→ Semantic model / Power BI reports / Copilot narratives

 

You do not need to move everything to Warehouse just to use Power BI or Copilot narratives. Copilot narratives work from Power BI reports and visuals, so the important part is having a good semantic model/report layer, not necessarily whether the Gold data physically sits in a Lakehouse or Warehouse. The Narrative visual with Copilot can summarize report pages, visuals, or report context.

 

References:
https://learn.microsoft.com/en-us/fabric/fundamentals/decision-guide-lakehouse-warehouse
https://learn.microsoft.com/en-us/fabric/fundamentals/decision-guide-data-store
https://learn.microsoft.com/en-us/power-bi/create-reports/copilot-create-narrative

 

For F2 specifically, I would keep the architecture simple at first. F2 is good for learning, prototypes, and small workloads, but you should monitor capacity usage carefully, especially if you add notebooks, pipelines, Warehouse queries, semantic models, and Copilot/AI usage.

 

A practical naming approach:
If you expect this to grow beyond only product data, use a broader domain name instead, here is a table of naming recommendation, take inspiration from it or just use them:

Workspace:
ws-<domain>-<env>

Lakehouses / Warehouses:
lh_silver_<domain>
lh_gold_<domain>
wh_gold_<domain>
 
Pipelines / notebooks:
pl_ingest_api_product
pl_silver_transform_product
pl_load_gold_product
 
Tables:
silver.customer
gold.dim_customer
 
My recommendation for your starting point:
1. Use one workspace for the product/domain, not one workspace named after a single report.
2. Keep Silver in Lakehouse.
3. Use Gold Lakehouse for Python/forecasting outputs.
4. Use Gold Warehouse only where you need curated SQL marts for reporting.
5. Build Power BI semantic models on top of the Gold layer.
6. Use Copilot narratives in the report layer, not as the main reason to choose Lakehouse vs Warehouse.
 
  • icassiem's avatar
    icassiem
    Post Prodigy
    1 month ago

    SoniAnkit , sannavajjala , Parchitect  Thank You very much

    Becuase copilot narratives is a big driver for reporting i need the arch to be designed to support this, the FC is low key might not happen but i want to ensure the design caters for this and possibly future domains

     

    1. Other domains i dont knw how this will grow and hence why im unsure to name the workspace for a domain of "Porduct", i dont know but will this be a bad design ifa future request from another domain requested help on reporting, so im thinking keeping it broad but then again if it does not evolve it sounds confusing - what do you suggest?

     

    2. i would like to have kept it lightwieght of everything in lakehouse but having gold lakhouse pluss whs means theres 3 layers with silver and i mostly need whs for the sql transformation or prep the silver into dim/facts?

     

    3. my other issue is i cant use data factory to tranform json semi i need to use python or ?

     

    4. Do in in silver/gold have a item per domain like silver_product and silver_IT ?

     

    5. I received push back regarding middleware like datafactory/python, when there are updates or patches does this impact my pipes code and needs updating too?

     

    Please Help

    • Parchitect's avatar
      Parchitect
      Solution Sage
      1 month ago
      Hi icassiem,

       

      I would keep this post focused on the medallion design. For pipeline patching/maintenance, I suggest opening a separate post so this thread does not become too broad.

       

      For your setup on F2, I would keep it simple:
      API / JSON / CSV
      → Silver Lakehouse
      → Gold Warehouse
      → Semantic model
      → Power BI reports / Copilot narratives

       

      1. Workspace naming
      If you are not sure whether this will stay Product-only, I would avoid naming the workspace too narrowly.

       

      Use something broader like:
      ws-analytics-dev
      ws-analytics-prod
      or:
      ws-commercial-analytics-dev
      ws-commercial-analytics-prod

       

      Then keep the domain in table/item names, for example:
      silver_product_customer
      silver_product_sales
      gold_dim_customer
      gold_fact_sales

       

      If Product or IT grows later, then split into separate domain workspaces.

       

      2. Gold Lakehouse + Gold Warehouse?
      You do not need both from day one.

       

      Start with:
      Silver Lakehouse = API, JSON, semi-structured data, Python/Spark transformations 
      Gold Warehouse = SQL marts, dim/fact tables, reporting-ready layer
      Only add a Gold Lakehouse later if you need Python/forecasting outputs stored as curated Delta tables.

       

      3. JSON / semi-structured data
      For simple JSON, Dataflow Gen2 may be enough.
      For complex JSON, nested arrays, API pagination, or schema drift, use a Fabric Notebook with Python/PySpark and write the result to the Silver Lakehouse.

       

      4. Domain split
      Do not create too many Lakehouses at the start.

       

      Start simple:
      lh_silver
      wh_gold

       

      Split by domain later only if you need different ownership, security, lifecycle, or performance boundaries.

       

      Short recommendation:
      - Silver = Lakehouse
      - Gold reporting marts = Warehouse
      - Forecasting/Python output = optional Gold Lakehouse later
      - Copilot narratives depend more on the semantic model/report design than whether Gold is Lakehouse or Warehouse
      • icassiem's avatar
        icassiem
        Post Prodigy
        1 month ago

        Parchitect Thank You

        1. Workspace = "Analytics"

        2. Lkh Silver + Whs Gold = "Domain" Product / IT etc

        3. Add Lhs Gold when FC needs, is it then still a medallion if lh_silver, wh_gold + lh_gold = 3 layers actually 4 if silver adds?

        4. Does this mean "If Product or IT grows later, then split into separate domain workspaces."  that under ws "analytics" i have lh_silver_Domain or should i only build what the need is now meaning i know its product iam guessing it could eveolve assumption, should i just focus on what is fact, like WS Product Analytics, lh_gol_tbl? How are arch criteria defined i guess what im lpoking for fact or possible assumption but its uneccessary broad or just push back when the need arrives as not part of design but then its not enterprise?

         

        Apologies for the silly questions