Forum Discussion
Handling version controlled Metadata files in Fabric
We use metadata driven configuration json files and we save them in git. To deploy the files to lakehouse we use fabric cli library as part of our CI CD process.
Hi ati_puri ,
Thanks for reaching out to Microsoft Fabric Community.
When syncing a Lakehouse with Git, only artifact metadata is pushed, not the data layer or configuration files. To manage JSON configs, store them as .json files in Git for version control and CI/CD support.Additionally, many teams use one or more of the following methods, depending on the configuration type:
Azure Key Vault / App Config: For managing secrets, credentials, or secure settings that vary by environment, accessible at runtime by pipelines and notebooks.
Pipeline/global parameters: For environment-specific values like paths or schema names, which can be dynamically passed into pipelines and notebooks.
Dedicated Config Lakehouse with deployment step: For configurations that must remain as files, .json files are maintained in Git and copied to a Config Lakehouse folder during deployment.
Typically, a hybrid model is adopted, using Git for structural or lookup metadata, Key Vault or pipeline parameters for sensitive and environment-specific settings, and an optional Config Lakehouse for file-based configurations within Fabric. This approach ensures both version control and flexibility when promoting solutions across Dev, Test, and Prod environments.