Microsoft Fabric - Designing a Medallion Architecture

[Poweraegg]

In this video we explore the medallion architecture inside Microsoft Fabric, and one possible way to setup the architecture using workspaces. This structure would ensure high data quality, the ability to manage data inside a CI/CD deployment pipeline and to ensure the highest quality of data possible for all stakeholders.

There are several ways you can structure the medallion architecture. For example you might want to use folders inside a workspace to seperate silver and gold workspace artifacts, and have less workspaces. You might also use domains to further containerize your data. Whichever the choice, the medallion architecture should be accompanied with a deployment pipeline to ensure integrity and quality. 

[JB81]

Hi

If your bronze layer uses software that has to be developed - doesn't it need testing before going live?

How do you know you are not going to take out the entire data engineering layer or kill other processes?

Interesting idea, not sure how that passes audit...

Thanks

JB

[Poweraegg]

Hi @JB81,

The silver layer is part of a deployment pipeline with 3 stages (dev-test-prod), with each stage having a designated workspace. The reason the bronze layer is not in a deployment pipeline is because it consists of the raw data layer, without any artifacts being developed or deployed, except for the bronze lakehouse. However should you chose to have any items in the bronze layer, you can of course create 2-3 bronze layer workspaces and link the to the deployment pipeline stages. This is up to you. There are many variations I will continue to discuss in the future. You can for example simply organize medallion layers in folders within a workspace, unifying bronze and silver in workspaces. 

Best regards,

Poweraegg