Medallion Archtecture

[icassiem]

Good day,

Kindly advise on what the modern archietecture for a fabric medallion of dilver and gold only, currently reading API & possibly semi structured data too

so my research says silver is my lakehouse delta db, but for gold do i contunre with a new db but for currated gold to have like dw marts for product and possible future forecasting with python

1. Can i move Gold to Fabric DW for my marts, sql, forecast write tables and most benefit i want is the copilot ai narratives, or best to stick with Gold in Lakehouse detla db?

2. Naming conventions, currently its for product and with code and stored in sharepoint , i only have an F2 subscription, Do i name the workspace the product but that limits me to only have product data in lakehouse and not future other data and then i tlooks funny it IT data in a Product workspace name, and do we call the workspace X? with silver lakhouse X? and gold lakehouse or DW what name?

Please advise, what is best paractices and methods to do and use please?

Regards

[SoniAnkit]

Hi @icassiem,

You are on the right track. In Microsoft Fabric, a common approach is to keep the Silver layer in a Lakehouse using Delta tables, especially for API and semi-structured data.

For the Gold layer, you can move to a Fabric Data Warehouse if your goal is to build curated data marts, support SQL analytics, create forecasting tables, and leverage Copilot capabilities. Many organizations use a hybrid approach:

• Silver: Lakehouse (Delta)

• Gold: Fabric Data Warehouse

For naming conventions, I recommend organizing workspaces by business domain or product area rather than by medallion layer. For example:

Workspace: ProductAnalytics

• lh_product_silver

• dw_product_gold

• semantic_product_sales

• nb_product_forecasting

This provides flexibility for future expansion and avoids creating too many workspaces, which is especially important with an F2 capacity.

For your use case (API ingestion, semi-structured data, analytics, and forecasting), the following architecture works well:

API/Semi-structured Data -> Silver Lakehouse -> Gold Fabric Data Warehouse -> Power BI/Copilot/Python Forecasting

This approach aligns with current Microsoft Fabric best practices and provides good scalability for future requirements.

Regards,
Ankit

[sannavajjala]

Hi,

The answer you received is solid, and I’d just reinforce one key point from a practical architecture perspective: in Fabric, you don’t need to force a single choice between Lakehouse and Warehouse for your Gold layer , it’s very common (and recommended) to use both depending on the workload. Keeping Silver in a Lakehouse is absolutely the right approach for API and semi-structured ingestion, and for Gold you can split responsibilities: use Gold Lakehouse for curated Delta tables, Python-based transformations, and forecasting scenarios, and use Gold Warehouse only where you need structured SQL marts, dimensional models, or BI-friendly serving layers. For Copilot narratives specifically, what matters is your semantic model and Power BI layer, not whether the data sits in Lakehouse or Warehouse. On naming and workspace design, avoid naming based on a single use case like “Product” , instead use a broader domain-based naming convention (e.g., per business domain) and keep items clearly structured (silver/gold separation at item level, not workspace per layer). Given you’re on F2, keep things simple initially (one workspace per domain, minimal duplication), and only introduce Warehouse or additional layers when you have a clear consumption need.

[Parchitect]

Hi  , 

So short answer:

- Yes, you can move Gold marts to Fabric Warehouse.

- But keep forecasting / Python-friendly curated tables in Gold Lakehouse if needed.

- For reporting, SQL marts, dimensional modelling, and BI consumption, Gold Warehouse is a good choice.

- For F2, keep it simple and avoid creating too many separate workspaces/items too early.

Longer answer:
For your scenario, I would not think of this as “Lakehouse OR Warehouse”. In Fabric it is very common to use both, depending on the purpose of the layer.

For your case:

- Silver = Lakehouse

- Gold = Lakehouse and/or Warehouse depending on consumption pattern

Since you are ingesting APIs and possibly semi-structured data, Silver in a Lakehouse is the right place. Lakehouse is better suited for Spark/Python, semi-structured data, data engineering, and future ML/forecasting workloads. Microsoft’s decision guide also points Lakehouse toward Spark, mixed/semi-structured data, big data, and machine learning scenarios.

For Gold, I would use this rule:

Use Gold Lakehouse when:

- you still need Spark/Python transformations

- you want to keep Delta tables for data science / forecasting

- the data is still evolving

- you want to train or score models with notebooks

Use Fabric Warehouse when:

- the data is structured and business-ready

- you want SQL-based marts

- you want star schemas / dimensional models

- your consumers are BI users and SQL users

- you need stronger SQL modelling patterns and curated serving tables

So a good pattern would be:

API / semi-structured source

→ Silver Lakehouse: cleaned, standardized, deduplicated Delta tables

→ Gold Lakehouse: curated analytical Delta tables and forecasting outputs

→ Gold Warehouse: SQL marts / dimensional model for reporting and business consumption

→ Semantic model / Power BI reports / Copilot narratives

You do not need to move everything to Warehouse just to use Power BI or Copilot narratives. Copilot narratives work from Power BI reports and visuals, so the important part is having a good semantic model/report layer, not necessarily whether the Gold data physically sits in a Lakehouse or Warehouse. The Narrative visual with Copilot can summarize report pages, visuals, or report context.

References:

https://learn.microsoft.com/en-us/fabric/fundamentals/decision-guide-lakehouse-warehouse

https://learn.microsoft.com/en-us/fabric/fundamentals/decision-guide-data-store

https://learn.microsoft.com/en-us/power-bi/create-reports/copilot-create-narrative

For F2 specifically, I would keep the architecture simple at first. F2 is good for learning, prototypes, and small workloads, but you should monitor capacity usage carefully, especially if you add notebooks, pipelines, Warehouse queries, semantic models, and Copilot/AI usage.

A practical naming approach:

If you expect this to grow beyond only product data, use a broader domain name instead, here is a table of naming recommendation, take inspiration from it or just use them:

Workspace:

ws-<domain>-<env>

Lakehouses / Warehouses:

lh_silver_<domain>

lh_gold_<domain>

wh_gold_<domain>

 

Pipelines / notebooks:

pl_ingest_api_product

pl_silver_transform_product

pl_load_gold_product

 

Tables:

silver.customer

gold.dim_customer

 

My recommendation for your starting point:

1. Use one workspace for the product/domain, not one workspace named after a single report.

2. Keep Silver in Lakehouse.

3. Use Gold Lakehouse for Python/forecasting outputs.

4. Use Gold Warehouse only where you need curated SQL marts for reporting.

5. Build Power BI semantic models on top of the Gold layer.

6. Use Copilot narratives in the report layer, not as the main reason to choose Lakehouse vs Warehouse.

[icassiem]

@SoniAnkit , @sannavajjala , @Parchitect  Thank You very much

Becuase copilot narratives is a big driver for reporting i need the arch to be designed to support this, the FC is low key might not happen but i want to ensure the design caters for this and possibly future domains

1. Other domains i dont knw how this will grow and hence why im unsure to name the workspace for a domain of "Porduct", i dont know but will this be a bad design ifa future request from another domain requested help on reporting, so im thinking keeping it broad but then again if it does not evolve it sounds confusing - what do you suggest?

2. i would like to have kept it lightwieght of everything in lakehouse but having gold lakhouse pluss whs means theres 3 layers with silver and i mostly need whs for the sql transformation or prep the silver into dim/facts?

3. my other issue is i cant use data factory to tranform json semi i need to use python or ?

4. Do in in silver/gold have a item per domain like silver_product and silver_IT ?

5. I received push back regarding middleware like datafactory/python, when there are updates or patches does this impact my pipes code and needs updating too?

Please Help

[Parchitect]

Hi @icassiem,

I would keep this post focused on the medallion design. For pipeline patching/maintenance, I suggest opening a separate post so this thread does not become too broad.

For your setup on F2, I would keep it simple:

API / JSON / CSV

→ Silver Lakehouse

→ Gold Warehouse

→ Semantic model

→ Power BI reports / Copilot narratives

1. Workspace naming

If you are not sure whether this will stay Product-only, I would avoid naming the workspace too narrowly.

Use something broader like:

ws-analytics-dev

ws-analytics-prod

or:

ws-commercial-analytics-dev

ws-commercial-analytics-prod

Then keep the domain in table/item names, for example:

silver_product_customer

silver_product_sales

gold_dim_customer

gold_fact_sales

If Product or IT grows later, then split into separate domain workspaces.

2. Gold Lakehouse + Gold Warehouse?

You do not need both from day one.

Start with:

Silver Lakehouse = API, JSON, semi-structured data, Python/Spark transformations 

Gold Warehouse = SQL marts, dim/fact tables, reporting-ready layer

Only add a Gold Lakehouse later if you need Python/forecasting outputs stored as curated Delta tables.

3. JSON / semi-structured data

For simple JSON, Dataflow Gen2 may be enough.

For complex JSON, nested arrays, API pagination, or schema drift, use a Fabric Notebook with Python/PySpark and write the result to the Silver Lakehouse.

4. Domain split

Do not create too many Lakehouses at the start.

Start simple:

lh_silver

wh_gold

Split by domain later only if you need different ownership, security, lifecycle, or performance boundaries.

Short recommendation:

- Silver = Lakehouse

- Gold reporting marts = Warehouse

- Forecasting/Python output = optional Gold Lakehouse later

- Copilot narratives depend more on the semantic model/report design than whether Gold is Lakehouse or Warehouse

[icassiem]

@Parchitect Thank You

1. Workspace = "Analytics"

2. Lkh Silver + Whs Gold = "Domain" Product / IT etc

3. Add Lhs Gold when FC needs, is it then still a medallion if lh_silver, wh_gold + lh_gold = 3 layers actually 4 if silver adds?

4. Does this mean "If Product or IT grows later, then split into separate domain workspaces."  that under ws "analytics" i have lh_silver_Domain or should i only build what the need is now meaning i know its product iam guessing it could eveolve assumption, should i just focus on what is fact, like WS Product Analytics, lh_gol_tbl? How are arch criteria defined i guess what im lpoking for fact or possible assumption but its uneccessary broad or just push back when the need arrives as not part of design but then its not enterprise?

Apologies for the silly questions

[Parchitect · Accepted solution]

Hi @icassiem,

No silly questions at all. These are exactly the right questions when you start thinking about data architecture instead of only solving one report.

The goal is to avoid a spaghetti architecture later.

I would separate two things:

1. What you know today

2. What might happen later

Do not over-design only based on assumptions. But also do not name everything so narrowly that it becomes hard to grow.

For your case, I would choose a balanced approach:

Workspace:

ws-analytics-dev

ws-analytics-prod

Then keep the domain in the item/table names:

lh_silver

wh_gold

Tables or schemas:

silver_product_customer

silver_product_sales

gold_dim_customer

gold_fact_sales

If Product becomes big enough later, or IT/Finance needs different ownership/security/lifecycle, then split into domain workspaces later:

ws-product-analytics-prod

ws-it-analytics-prod

So yes, start with what is factual today, but use naming that does not block future growth.

For the medallion question:

Yes, it is still medallion even if the physical items are different.

Medallion is a logical pattern, not a rule that every layer must be the same technology.

Example:

Silver = Lakehouse

Gold = Warehouse

Optional forecasting output = Gold Lakehouse or ML output table

That is still a medallion architecture because the logic is:

Source → cleaned/standardized → curated/business-ready

I would not create lh_gold unless you have a clear need for it.

For now, I would keep it simple:

API / JSON / CSV

→ lh_silver

→ wh_gold

→ semantic model

→ Power BI / Copilot narratives

Add lh_gold later only if forecasting/Python needs curated Delta output before loading to the Warehouse.

Architecture criteria should be based on:

- ownership

- security

- lifecycle

- performance

- data reuse

- maintainability

Not only “possible future request”.

[icassiem]

@Parchitect  wow thank you 😊

does this "ws-product-analytics-prod + ws-it-analytics-prod" mean 2 workspaces and F2 license wont work as i think it only allows for one workspace and then i will have different silv+gold arch correct? probably best just to keep it universal the 1 workspace like "silver_product_sales & silver_IT_storage"

[Parchitect]

No, F2 does not mean you can only have one workspace. You can have multiple workspaces assigned to the same Fabric capacity.

The important point is that they all share the same F2 compute. So technically you can have:

 ws-product-analytics-prod

ws-it-analytics-prod

on the same F2 capacity, but on a small F2 SKU I would only split workspaces when there is a real reason. 

Based on your situation, I agree with your thinking: keep it simple and use one broader workspace.

[icassiem]

@Parchitect  Thank You 🙏