topic Re: setup of medallion architecture in Data Engineering

setup of medallion architecture

shivam_agarwal — Mon, 25 Dec 2023 04:49:05 GMT

Hi all, I'm relatively new to MIcrosoft Fabric. I am going through the docs but as a beginner, it is really getting overwhelming.I wanted to know what is the step by step process to setup a medallion architecture?

Re: setup of medallion architecture

govindarajan_d — Mon, 25 Dec 2023 10:06:09 GMT

Hi @shivam_agarwal ,

There are multiple ways you can setup Medallion architecture:

Using Lakehouses/Warehouses and each zone (bronze, silver and gold) in separate workspace. This is advantageous as you can separate the access based on workspace. But I found it to be effective only if I use entirely Lakehouse or Warehouse. If you need Spark, then you can go entirely with Lakehouse or else just Warehouse if you plan to use plain T-SQL. This boils down to the skillset of the team.

The other approach would be having all three layers in same workspace or just silver and gold in same workspace so that you can mix Lakehouse/Warehouse very easily as both silver and gold are in the same workspace.

For a more step-by-step process, follow this exercise:

mslearn-fabric (microsoftlearning.github.io)

Re: setup of medallion architecture

AndyDDC — Mon, 25 Dec 2023 12:12:18 GMT

I have a part 1 in a small series where I'm looking at medallion architecture and what data loading options there are

https://www.serverlesssql.com/lakehouse-zones/

Re: setup of medallion architecture

jakes — Mon, 04 Nov 2024 19:10:39 GMT

Andy - great stuff in your post. But I can't find Part 2. Can you give me the link?