Forum Discussion

nomad_ch's avatar
nomad_ch
New Member
10 months ago
Solved

Medallion architecture with CI/CD Pipelines

Hello,

 

I’m an analyst transitioning to data engineering and I am quite new to Fabric. I am learning how to structure a medallion architecture (Bronze → Silver → Gold) in Microsoft Fabric across Dev, Test, and Prod environments.

 

I want to set up my OneLake and warehouse/lakehouse correctly before building pipelines. I’m thinking of this structure:

 

- OneLake (staging area for raw files, e.g., Files/raw/)

 

- Three workspaces for environments:

- Dev_Workspace -> Warehouse or Lakehouse with BronzeSchema, SilverSchema, GoldSchema

- Test_Workspace -> Warehouse or Lakehouse with BronzeSchema, SilverSchema, GoldSchema

- Prod_Workspace -> Warehouse or Lakehouse with BronzeSchema, SilverSchema, GoldSchema

 

Does this structure make sense for Fabric?

Is Bronze needed at all in Dev/Test for testing ingestion from OneLake? 

Any tips for CI/CD with this setup?

Thanks!

data engineering
data pipeline

5 Replies

  • tayloramy's avatar
    tayloramy
    Super User
    10 months ago

    Hi nomad_ch, 

     

    This setup makes sense to me. 

    I do a silimar thing, using a Lakehouse as my bronze and silver layers, and then I use a warehouse as my gold layer so I can apply RLS to it. 

     

    as for CI/CD, what I recommend is connecting GitHub or Azure DevOps to your dev workspace for version control, and then use deployment pipelines to move items from dev up to test and then to prod. 

     

     

    There's an accelerator in the Fabric Toolbox for this: fabric-toolbox/accelerators/CICD/Deploy-using-Fabric-deployment-pipelines at main · microsoft/fabric-toolbox

     

    If you found this helpful, consider giving some Kudos. If I answered your question or solved your problem, mark this post as the solution.

     

  • nielsvdc's avatar
    nielsvdc
    Super User
    10 months ago

    Hi nomad_ch,

     

    If you are just starting out and want to keep it simpel and small, 3 workspaces is good enough. Depending on the processing workload on dev-test and production, you might also want to think about a production capacity and a dev-test capacity, so that any development and test processes won't burn out your ability to run production due to CU shortage.

     

    Instead of developing a framework yourself, you can also check the ready-to-use metadata driven framework from Erwin de Kreuk on Github https://github.com/edkreuk/FMD_FRAMEWORK/.

     

    Hope this helps. If so, please give a Kudos 👍 and mark as Accepted Solution ✔️.

  • v-dineshya's avatar
    v-dineshya
    Community Support
    10 months ago

    Hi nomad_ch ,

    Thank you for reaching out to the Microsoft Community Forum.

     

    Hi tayloramy  and nielsvdc , Thank you for your prompt responses.

     

    Hi nomad_ch ,  Could you please try the proposed solutions shared by tayloramy  and nielsvdc ? Let us know if you’re still facing the same issue we’ll be happy to assist you further.

     

    Regards,

    Dinesh

    • v-dineshya's avatar
      v-dineshya
      Community Support
      9 months ago

      Hi nomad_ch ,

      We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. And, if you have any further query do let us know.

       

      Regards,

      Dinesh

  • KevinChant's avatar
    KevinChant
    Super User
    10 months ago

    This looks fine to me. in addition to the previous framework shared you can also look to use Fabric Accelerator which roots can be traced back to offerings for other services:

    https://github.com/bennyaustin/fabric-accelerator 

    For CI/CD across traditional DTAP environments you might also want to look into using the fabric-cicd Python library with your ALM service of choice.
    https://microsoft.github.io/fabric-cicd/latest/ 

    I have a fair few posts relating to performing CI/CD in Fabric which might help:
    https://www.kevinrchant.com/?s=fabric+ci%2Fcd