Git integration for Datawarehouse

[caseybks]

Hi,

I'm using the git integration within a fabric workspace, together with fabric-cicd to deploy code across workspaces which will form our dev, test and prod environments. From our develop branch, which is our collaboration branch that the team branch off and merge into, we are branching off into our own feature dev workspaces. 

I am finding that when we make an update to an existing table in the datawarehouse, e.g. add column or delete column, when the change is merged from a feature branch to develop branch, therefore updating the developement workspace env, this wipes the data from a table. Is there a way for this not to happen? I have tried updating schema via VS Code too, but this still has the same effect when the development workspace syncs up with the updated develop branch.

I'm assuming this happens for the SQL databases too.

Thanks

[v-tsaipranay · Accepted solution]

Hi @caseybks ,

Thank you for reaching out to the Microsoft fabric community forum and for sharing the details of your scenario. Also thank you @Arul  for your helplful response.

I understand that you are using Git integration with Fabric CI/CD for Data Warehouse deployments and experiencing data loss in tables when schema changes, such as adding or removing columns, are merged from a feature branch into the develop branch and then synced to the development workspace. This occurs because, due to current limitations in Fabric Git integration for Data Warehouses, schema changes made through Git sync will recreate the underlying object to match the repository definition, which results in the data being cleared.

This limitation, which also applies to SQL databases in Fabric, is documented here: Source control in Fabric Data Warehouse – Limitations.

To address this, you can back up your data before making schema changes by using COPY INTO or exporting to staging or external storage, then reload the data after deployment. Keep schema and data operations separate by applying schema updates via Git and restoring data with post-deployment scripts in Fabric pipelines or Fabric-CICD. It's important to test these steps in a staging environment before moving to development or production. Since Git sync can't currently preserve table data during schema changes, these methods can help manage the impact until the platform provides a solution.

Hope this helps. Please reach out for further assistance.

Thank you.

[Arul]

Hi @caseybks ,

As mentioned in the Microsoft documentation, this limitation is expected, and we now need to look for a manual workaround.

Please refer the limitation section of this documentation: https://learn.microsoft.com/en-us/fabric/data-warehouse/source-control?