Medallion architecture with CI/CD Pipelines

[nomad_ch]

Hello,

I’m an analyst transitioning to data engineering and I am quite new to Fabric. I am learning how to structure a medallion architecture (Bronze → Silver → Gold) in Microsoft Fabric across Dev, Test, and Prod environments.

I want to set up my OneLake and warehouse/lakehouse correctly before building pipelines. I’m thinking of this structure:

- OneLake (staging area for raw files, e.g., Files/raw/)

- Three workspaces for environments:

- Dev_Workspace -> Warehouse or Lakehouse with BronzeSchema, SilverSchema, GoldSchema

- Test_Workspace -> Warehouse or Lakehouse with BronzeSchema, SilverSchema, GoldSchema

- Prod_Workspace -> Warehouse or Lakehouse with BronzeSchema, SilverSchema, GoldSchema

Does this structure make sense for Fabric?

Is Bronze needed at all in Dev/Test for testing ingestion from OneLake? 

Any tips for CI/CD with this setup?

Thanks!

[tayloramy · Accepted solution]

Hi @nomad_ch, 

This setup makes sense to me. 

I do a silimar thing, using a Lakehouse as my bronze and silver layers, and then I use a warehouse as my gold layer so I can apply RLS to it. 

as for CI/CD, what I recommend is connecting GitHub or Azure DevOps to your dev workspace for version control, and then use deployment pipelines to move items from dev up to test and then to prod. 

There's an accelerator in the Fabric Toolbox for this: fabric-toolbox/accelerators/CICD/Deploy-using-Fabric-deployment-pipelines at main · microsoft/fabric...

If you found this helpful, consider giving some Kudos. If I answered your question or solved your problem, mark this post as the solution.

[KevinChant · Accepted solution]

This looks fine to me. in addition to the previous framework shared you can also look to use Fabric Accelerator which roots can be traced back to offerings for other services:

https://github.com/bennyaustin/fabric-accelerator 

For CI/CD across traditional DTAP environments you might also want to look into using the fabric-cicd Python library with your ALM service of choice.
https://microsoft.github.io/fabric-cicd/latest/ 

I have a fair few posts relating to performing CI/CD in Fabric which might help:
https://www.kevinrchant.com/?s=fabric+ci%2Fcd 

[v-dineshya]

Hi @nomad_ch ,

Thank you for reaching out to the Microsoft Community Forum.

Hi @tayloramy  and @nielsvdc , Thank you for your prompt responses.

Hi @nomad_ch ,  Could you please try the proposed solutions shared by @tayloramy  and @nielsvdc ? Let us know if you’re still facing the same issue we’ll be happy to assist you further.

Regards,

Dinesh

[v-dineshya]

Hi @nomad_ch ,

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. And, if you have any further query do let us know.

Regards,

Dinesh

[nielsvdc]

Hi @nomad_ch,

If you are just starting out and want to keep it simpel and small, 3 workspaces is good enough. Depending on the processing workload on dev-test and production, you might also want to think about a production capacity and a dev-test capacity, so that any development and test processes won't burn out your ability to run production due to CU shortage.

Instead of developing a framework yourself, you can also check the ready-to-use metadata driven framework from Erwin de Kreuk on Github https://github.com/edkreuk/FMD_FRAMEWORK/.

Hope this helps. If so, please give a Kudos 👍 and mark as Accepted Solution ✔️.

[tayloramy · Accepted solution]

Hi @nomad_ch, 

This setup makes sense to me. 

I do a silimar thing, using a Lakehouse as my bronze and silver layers, and then I use a warehouse as my gold layer so I can apply RLS to it. 

as for CI/CD, what I recommend is connecting GitHub or Azure DevOps to your dev workspace for version control, and then use deployment pipelines to move items from dev up to test and then to prod. 

There's an accelerator in the Fabric Toolbox for this: fabric-toolbox/accelerators/CICD/Deploy-using-Fabric-deployment-pipelines at main · microsoft/fabric...

If you found this helpful, consider giving some Kudos. If I answered your question or solved your problem, mark this post as the solution.