How implementing the medallion architecture in Fabric

[pmscorca]

Hi,

in Fabric how is it possible to implement the medallion architecture?

Are there more approaches?

Thanks

[v-cboorla-msft · Accepted solution]

Hi @pmscorca 

Thanks for using Microsoft Fabric Community.

The medallion lakehouse architecture, commonly known as medallion architecture, is a design pattern that's used by organizations to logically organize data in a lakehouse. It's the recommended design approach for Fabric.

Medallion architecture comprises three distinct layers—or zones. Each layer indicates the quality of data stored in the lakehouse, with higher levels representing higher quality. This multi-layered approach helps you to build a single source of truth for enterprise data products.

Importantly, medallion architecture guarantees the ACID set of properties (Atomicity, Consistency, Isolation, and Durability) as data progresses through the layers. Starting with raw data, a series of validations and transformations prepares data that's optimized for efficient analytics. There are three medallion stages: bronze (raw), silver (validated), and gold (enriched).

To implement medallion architecture in Fabric, you can either use lakehouses (one for each zone), a data warehouse, or combination of both. Your decision should be based on your preference and the expertise of your team. Keep in mind that Fabric provides you with flexibility: You can use different analytic engines that work on the one copy of your data in OneLake.

Here are two patterns to consider.

• Pattern 1: Create each zone as a lakehouse. In this case, business users access data by using the SQL analytics endpoint.
• Pattern 2: Create the bronze and silver zones as lakehouses, and the gold zone as data warehouse. In this case, business users access data by using the data warehouse endpoint.

For additional information please refer to the below documentations:

Link 1 : Implement medallion lakehouse architecture in Microsoft Fabric

Link 2 : Medallion architecture in Microsoft Fabric

I hope this information helps. Please do let us know if you have any further queries.

Thank you.

[pmscorca]

Hi, thanks for your reply.

It is recommended to create a workspace for each lakehouse/zone, isn't it?

[v-cboorla-msft]

Hi @pmscorca 

Yes, It is recommended to create each lakehouse in its own, separate Fabric workspace. This approach provides you with more control and better governance at the zone level. In the context of the medallion architecture for a data lakehouse on Microsoft Fabric, creating a separate workspace for each zone (bronze, silver, gold) is considered a best practice.

Reason for separating lakehouses by workspace is beneficial because of:

Data Governance and Isolation: Separate workspaces enforce clear boundaries between data at different stages of refinement. This improves data governance by restricting access and modifications based on user roles and zone permissions.

Improved Lineage Tracking: With distinct workspaces, it's easier to track the lineage of data across zones. You can easily identify the origin and transformations applied to data as it flows through the pipeline.

Optimized Resource Management: Each zone likely has different performance and storage needs. Separate workspaces allow for independent resource allocation, ensuring optimal resource utilization for each zone.

I hope this information helps. Please do let us know if you have any further queries.

Thank you.

[v-cboorla-msft]

Hi @pmscorca 

Glad that you query got resolved.
Please continue using Fabric Community for further queries.

Thanks.