Forum Discussion
SIlver Layer - Additional Columns
- 11 months ago
Hi michaelgambling ,
Thank you for reaching out to Microsoft Fabric Community.
Thank you Vinodh247 for the prompt response.
Here are few documentations that will help:
1.Medallion Lakehouse Architecture in Fabric (Bronze/Silver/Gold)
Implement medallion lakehouse architecture in Fabric - Microsoft Fabric | Microsoft Learn
2.Lakehouse End-to-End Tutorial (Fabric)
Lakehouse end-to-end scenario: overview and architecture - Microsoft Fabric | Microsoft Learn
3.Handle Schema Drift (Dataflow Gen2)
How to handle schema drift in Dataflow Gen2 - Microsoft Fabric | Microsoft Learn
4.Schema Drift Concepts (Mapping Dataflows)
Schema drift in mapping data flow - Azure Data Factory & Azure Synapse | Microsoft Learn
5.Lab Exercise: Build Medallion Lakehouse
Create a medallion architecture in a Microsoft Fabric lakehouse | mslearn-fabric
6.Incremental Copy Patterns (Watermarking)
Incrementally load data from Data Warehouse to Lakehouse - Microsoft Fabric | Microsoft Learn
This is a common point of confusion when designing Bronze/Silver/Gold layers in fabric. The best practice is to treat the silver layer as a curated, standardized, business ready data layer that sits between your raw bronze and consumption focused gold. That means the transformations, enrichment, and the addition of new columns (for business logic, calculated fields, derived metrics, or standardized formats) should happen as part of your data eng pipeline, not left to PBI engineers. If you allow PBI engineers to add columns directly in the Silver lakehouse, you risk losing consistency, governance, and repeatability because their changes will not survive pipeline refreshes. To avoid overwriting issues, design your Bronze-to-Silver pipeline to be idempotent and schema-aware: either handle schema drift (autodetect and merge new columns) or manage transformations centrally in notebooks or dataflows that explicitly add those additional columns during the load. That way, your Silver tables become the single, trusted layer that always exposes the right schema, while PBI engineers focus on modeling and visualization in the semantic layer. In short, keep schema evolution and column additions inside your pipeline, not ad hoc in PBI or manual edits in the lakehouse.
- michaelgambling11 months agoAdvocate I
Thanks for the freedback, is there any documentation that might assist in how to set something like this up? Im a Sys Ad, i have no background in data engineering, anything you can siggest article wise that may help me make sense of this would be greatly appreciated, thanks so much fo ryour feedback!