Forum Discussion
Medallion Archtecture
- Hi icassiem,No silly questions at all. These are exactly the right questions when you start thinking about data architecture instead of only solving one report.The goal is to avoid a spaghetti architecture later.I would separate two things:1. What you know today2. What might happen laterDo not over-design only based on assumptions. But also do not name everything so narrowly that it becomes hard to grow.For your case, I would choose a balanced approach:Workspace:ws-analytics-devws-analytics-prodThen keep the domain in the item/table names:lh_silverwh_goldTables or schemas:silver_product_customersilver_product_salesgold_dim_customergold_fact_salesIf Product becomes big enough later, or IT/Finance needs different ownership/security/lifecycle, then split into domain workspaces later:ws-product-analytics-prodws-it-analytics-prodSo yes, start with what is factual today, but use naming that does not block future growth.For the medallion question:Yes, it is still medallion even if the physical items are different.Medallion is a logical pattern, not a rule that every layer must be the same technology.Example:Silver = LakehouseGold = WarehouseOptional forecasting output = Gold Lakehouse or ML output tableThat is still a medallion architecture because the logic is:Source → cleaned/standardized → curated/business-readyI would not create lh_gold unless you have a clear need for it.For now, I would keep it simple:API / JSON / CSV→ lh_silver→ wh_gold→ semantic model→ Power BI / Copilot narrativesAdd lh_gold later only if forecasting/Python needs curated Delta output before loading to the Warehouse.Architecture criteria should be based on:- ownership- security- lifecycle- performance- data reuse- maintainabilityNot only “possible future request”.
SoniAnkit , sannavajjala , Parchitect Thank You very much
Becuase copilot narratives is a big driver for reporting i need the arch to be designed to support this, the FC is low key might not happen but i want to ensure the design caters for this and possibly future domains
1. Other domains i dont knw how this will grow and hence why im unsure to name the workspace for a domain of "Porduct", i dont know but will this be a bad design ifa future request from another domain requested help on reporting, so im thinking keeping it broad but then again if it does not evolve it sounds confusing - what do you suggest?
2. i would like to have kept it lightwieght of everything in lakehouse but having gold lakhouse pluss whs means theres 3 layers with silver and i mostly need whs for the sql transformation or prep the silver into dim/facts?
3. my other issue is i cant use data factory to tranform json semi i need to use python or ?
4. Do in in silver/gold have a item per domain like silver_product and silver_IT ?
5. I received push back regarding middleware like datafactory/python, when there are updates or patches does this impact my pipes code and needs updating too?
Please Help
Parchitect Thank You
1. Workspace = "Analytics"
2. Lkh Silver + Whs Gold = "Domain" Product / IT etc
3. Add Lhs Gold when FC needs, is it then still a medallion if lh_silver, wh_gold + lh_gold = 3 layers actually 4 if silver adds?
4. Does this mean "If Product or IT grows later, then split into separate domain workspaces." that under ws "analytics" i have lh_silver_Domain or should i only build what the need is now meaning i know its product iam guessing it could eveolve assumption, should i just focus on what is fact, like WS Product Analytics, lh_gol_tbl? How are arch criteria defined i guess what im lpoking for fact or possible assumption but its uneccessary broad or just push back when the need arrives as not part of design but then its not enterprise?
Apologies for the silly questions
- Hi icassiem,No silly questions at all. These are exactly the right questions when you start thinking about data architecture instead of only solving one report.The goal is to avoid a spaghetti architecture later.I would separate two things:1. What you know today2. What might happen laterDo not over-design only based on assumptions. But also do not name everything so narrowly that it becomes hard to grow.For your case, I would choose a balanced approach:Workspace:ws-analytics-devws-analytics-prodThen keep the domain in the item/table names:lh_silverwh_goldTables or schemas:silver_product_customersilver_product_salesgold_dim_customergold_fact_salesIf Product becomes big enough later, or IT/Finance needs different ownership/security/lifecycle, then split into domain workspaces later:ws-product-analytics-prodws-it-analytics-prodSo yes, start with what is factual today, but use naming that does not block future growth.For the medallion question:Yes, it is still medallion even if the physical items are different.Medallion is a logical pattern, not a rule that every layer must be the same technology.Example:Silver = LakehouseGold = WarehouseOptional forecasting output = Gold Lakehouse or ML output tableThat is still a medallion architecture because the logic is:Source → cleaned/standardized → curated/business-readyI would not create lh_gold unless you have a clear need for it.For now, I would keep it simple:API / JSON / CSV→ lh_silver→ wh_gold→ semantic model→ Power BI / Copilot narrativesAdd lh_gold later only if forecasting/Python needs curated Delta output before loading to the Warehouse.Architecture criteria should be based on:- ownership- security- lifecycle- performance- data reuse- maintainabilityNot only “possible future request”.
Parchitect wow thank you 😊
does this "ws-product-analytics-prod + ws-it-analytics-prod" mean 2 workspaces and F2 license wont work as i think it only allows for one workspace and then i will have different silv+gold arch correct? probably best just to keep it universal the 1 workspace like "silver_product_sales & silver_IT_storage"