Forum Discussion

bertan's avatar
bertan
New Member
11 months ago
Solved

Implementing Medallion Architecture in Fabric with Data from Multiple Source Systems

Hi all,

 

I just started a new job and one of my current main responsibilities is to implement a medallion architecture to improve our processing of source data from our ERP systems into report-ready formats.

 

I want to extract data from 3-4 different ERP systems into a bronze layer using Data Factory pipelines into my lakehouse, but noticed quite early that it is in fact not possible to create subfolders for each data source in my lakehouse. Since there will be many tables from each source I would prefer a solid and scalable structure and avoid too much clutter.

 

I would like to have some guidance on how to structure the Medallion architecture for my use case. I read some other people's posts, such asMedallion Architecture - best practices to combine data sources , but don't see that they quite apply to my use case. I also thought of the solution to land all raw files into folders of the Files section of my bronze lakehouse, one for each source system. And then connect to these source files from the silver layer. Would this approach be recommended? Any other advice would be much appreicated. Thanks in advance! 

workspaces
  • FabVarunM's avatar
    FabVarunM
    11 months ago

    Hello bertan 

     

    I have been in similar situation  and being a Data Architect what i have implemented is, 
    I created a lakehouse with Lakehouse Schema as shown in below Picture, and then created Schema for each ERP system. Then I created a pipeline and while ingesting the data into Lakeshoue I defined where the table will be loaded, as per the screenshot. This helps me to organise my different sources at Bronze lakeshouse.  Later you can read them in Silver layer for Transformation and Gold for making it ready for Consumption.

    I hope  this will help you, and please accept the asnwer. Good luck in your new role I am sure you will thrive. Cheers! 🙂 

    Thanks,

    Varun

3 Replies

  • AntoineW's avatar
    AntoineW
    Super User
    11 months ago

    Hello bertan,

     

    Thanks for sharing your use case—it's a great scenario for applying the Medallion architecture in a scalable way.

    You're right, the Tables area is managed (Delta) and doesn’t support subfolders per source.

     

    However, there are a few best practices and workarounds you can apply to maintain a clean and scalable structure:

     

    ✅ Recommended Structure for Your Use Case

    1. Bronze Layer (Raw Ingestion)

    • Use Data Factory pipelines to land raw data into the Files section of your Lakehouse.
    • You can create a folder per ERP system in the Files part (e.g., /bronze/files/erp1/, /bronze/files/erp2/, etc.).
    • Store raw files in their native format (CSV, JSON, Parquet, etc.) this gives you flexibility and traceability.

    2. Silver Layer (Cleaned & Modeled)

    • Use Notebooks or Dataflows Gen2 to read from the raw files and transform them into structured tables.
    • Store these cleaned tables in the Tables section of a separate Lakehouse or in a dedicated schema (enable Schema when creating lakehouse).
    • You can prefix table names to reflect the source system, e.g., erp1_customers, erp2_orders, etc., to avoid clutter and maintain clarity.

    3. Gold Layer (Business-Ready)

    • Depending on the governance on Fabric on your company, you may store golden data in a different workspace (in a lakehouse/warehouse)
    • Using Notebook/Dataflow gen2 to aggregate and join data across sources to create business-friendly, report-ready tables.
    • Datacan be exposed in directLake into semantic model to serve Power BI for reporting.

     

    Hope it can help you,

    Best regards,

    Antoine

  • FabVarunM's avatar
    FabVarunM
    New Member
    11 months ago

    Hello bertan 

     

    I have been in similar situation  and being a Data Architect what i have implemented is, 
    I created a lakehouse with Lakehouse Schema as shown in below Picture, and then created Schema for each ERP system. Then I created a pipeline and while ingesting the data into Lakeshoue I defined where the table will be loaded, as per the screenshot. This helps me to organise my different sources at Bronze lakeshouse.  Later you can read them in Silver layer for Transformation and Gold for making it ready for Consumption.

    I hope  this will help you, and please accept the asnwer. Good luck in your new role I am sure you will thrive. Cheers! 🙂 

    Thanks,

    Varun

    • bertan's avatar
      bertan
      New Member
      11 months ago

      Thank you so much! This is much much appreciated! Exactly what I was looking for for the Silver layer.