Handling version controlled Metadata files in Fabric

[ati_puri]

Hi,

Currently, in our project we are handling configuration files for the metadata-driven pipelines/notebooks in the file section of fabric lakehouse, however when we sync our lakehosue with GIT, only lakehouse metadata goes into GIT and not datafiles. This leads to an issue when we want to have those config files in our other environments like Test, Prod via GIT.

One solution is to have those metadata files in GIT as .josn files, is there any other solution which is currently being implemented as a version controlled metadata files inside or outside fabric.

[gaya3krishnan86 · Accepted solution]

We use metadata driven configuration json files and we save them in git.  To deploy the files to lakehouse we use fabric cli library as part of our CI CD process. 

[ati_puri]

Has anyone tried Variable library for the above?

[v-sshirivolu]

Hi @ati_puri ,
Thanks for reaching out to Microsoft Fabric Community.

When syncing a Lakehouse with Git, only artifact metadata is pushed, not the data layer or configuration files. To manage JSON configs, store them as .json files in Git for version control and CI/CD support.Additionally, many teams use one or more of the following methods, depending on the configuration type:

Azure Key Vault / App Config: For managing secrets, credentials, or secure settings that vary by environment, accessible at runtime by pipelines and notebooks.

Pipeline/global parameters: For environment-specific values like paths or schema names, which can be dynamically passed into pipelines and notebooks.

Dedicated Config Lakehouse with deployment step: For configurations that must remain as files, .json files are maintained in Git and copied to a Config Lakehouse folder during deployment.

Typically, a hybrid model is adopted, using Git for structural or lookup metadata, Key Vault or pipeline parameters for sensitive and environment-specific settings, and an optional Config Lakehouse for file-based configurations within Fabric. This approach ensures both version control and flexibility when promoting solutions across Dev, Test, and Prod environments.