Warehouse, git integration and dbt

[Johanneszv]

Hi all,

TL;DR: Is it possible to disable tracking of changes in data warehouse files in Fabric’s Git integration? If so, how? We have tried .gitignore and Fabric does not hounor that. 

Context: 

We are using dbt for transformations for data stored in a Fabric Warehouse. We leverage Fabric’s Git integration to version-control other items and dbt code. Our workflow includes:

- A single production workspace synced with main and multiple feature/person-specific development workspaces each synced with a feature branch.

- A trunk-based Git model where pull requests are made from feature/personal workspaces into main.

- Syncing as the mechanism to move code between environments. Read: no deployment pipelines 

The issue arises when we run dbt build and create views, tables, etc., in the Data Warehouse. When we do this (either in production or in feature branch) Fabric generates files to maintain the state of the warehouse. These files are further tracked in Git. This is problematic for us because:

1.We want dbt to manage the state of views, tables, etc. 

2.We only want to version-control dbt code in Git—not Fabric’s data warehouse state files because we want to run dbt buid as part of production pipeline in ADO or other means.

Questions:

- Is there a way to prevent Fabric from tracking these Data Warehouse files in Fabric’s Git integration? 

- Alternatively, are there other workarounds to solve this with our setup?

Thanks in advance for your help!

Johannes

[DataBard · Accepted solution]

If the Fabric/Git integration does not support .gitignore, I'd expect that no, there is no way to prevent Fabric from tracking certain changes in the Warehouse. The integration looks at the warehouse as one singular object, so picking and choosing what is/is not tracked or deployed hasn't been a feature.

As for workarounds, I can think of two directions you could go depending on where you want to manage your deployment process:

1. Somewhere between your dev's branches and your trunk you'd have to create an approach that either cherry picks what you want to merge to trunk, or creates an intermediate branch that can then be modified before changes are merged into trunk.
2. If you're looking to source control dbt SQL files within your warehouse, you can use a Schema Compare in visual studio to compare between two warehouses and create deployment scripts. You'd then keep your dev/feature branches separate from your trunk branch, and your commit to the trunk branch would be from your Prod workspace rather than a merge from a feature branch.
   1. If you want to see what the schema compare looks like, I made a video on my Youtube channel on this approach recently. You can find it here: https://youtu.be/OWF6YlAAY54?si=AhLY9p9QfLx0puQe

I've heard there might be changes coming to the git integration and deployment process in the future, but no idea when.

Once you've found a suitable answer, be sure to select your answer so other community members can learn from it.

[Johanneszv]

Thanks all! 

@AndyDDC:

- Good idea. Yes, that will work. At least the warehouse won't be tracked in Git. Ideally we would like it all to stay in the same workspace, as it all belong to the same data platform. 

Is dbt running within a Notebook?  I assume that's how it's version controlled?

- The dbt project is checked into git (same repo as the fabric code) just like the jaffle shop example from dbt. 

- We run dbt build from an Azure Devops Pipeline, which sends of the sql statements to the Warehouse. So the orchestration it self is not currently handled inside Fabric.  

On that note, I see that dbt cli support for data factory is scheduled for Q1 2025. Given you work with Warehouse and that is the only (OLAP) service dbt is compatible with in Fabric, do you know anything about this upcomming feature? https://learn.microsoft.com/en-us/fabric/release-plan/data-factory#data-pipeline-support-dbt 

@v-huijiey-msft:

Thanks for clarification. The deletion of files or at least prevent these files from being merged to main is something we can automate with Azure Devops Pipelines. We will test this one out, thanks!

Thanks @DataBard. The first option you proposed is something i will test out. I'll also have a look at your youtube video, thanks!

[v-huijiey-msft]

Hi @Johanneszv ,

Thank you for your interest in this case.

Is my follow-up just to ask if the problem has been solved?

If so, can you accept the correct answer as a solution or share your solution to help other members find it faster?

Thank you very much for your cooperation!

Best Regards,
Yang
Community Support Team

If there is any post helps, then please consider Accept it as the solution  to help the other members find it more quickly.
If I misunderstand your needs or you still have problems on it, please feel free to let us know. Thanks a lot!

[AndyDDC]

Ah ok. So you're running Dbt cloud?

I don't know much about the Dbt integration in data factory other than someone in MS accidentally switched it on a few months ago but although there was a Dbt task, it didn't actually work.

hopefully we'll get some more info soon

[v-huijiey-msft · Accepted solution]

Hi @Johanneszv ,

Thanks for the reply from AndyDDC / DataBard .

Currently, Fabric does not provide a way to prevent tracking of Data Warehouse files in Git integrations.

You can manually delete unwanted files before committing to Git.

I understand that this can be cumbersome, but it's a workaround until there's a better solution.

If you have a better solution please share it here, it will be of great help to users who are experiencing similar issues.

Best Regards,
Yang
Community Support Team

If there is any post helps, then please consider Accept it as the solution  to help the other members find it more quickly.
If I misunderstand your needs or you still have problems on it, please feel free to let us know. Thanks a lot!

[AndyDDC]

Hi @Johanneszv the only way I could see this working is if the Warehouse was in a workspace not connected to Git.  I assume dbt handles all the schema changes regardless of environment? 

Is dbt running within a Notebook?  I assume that's how it's version controlled?

[DataBard · Accepted solution]

If the Fabric/Git integration does not support .gitignore, I'd expect that no, there is no way to prevent Fabric from tracking certain changes in the Warehouse. The integration looks at the warehouse as one singular object, so picking and choosing what is/is not tracked or deployed hasn't been a feature.

As for workarounds, I can think of two directions you could go depending on where you want to manage your deployment process:

1. Somewhere between your dev's branches and your trunk you'd have to create an approach that either cherry picks what you want to merge to trunk, or creates an intermediate branch that can then be modified before changes are merged into trunk.
2. If you're looking to source control dbt SQL files within your warehouse, you can use a Schema Compare in visual studio to compare between two warehouses and create deployment scripts. You'd then keep your dev/feature branches separate from your trunk branch, and your commit to the trunk branch would be from your Prod workspace rather than a merge from a feature branch.
   1. If you want to see what the schema compare looks like, I made a video on my Youtube channel on this approach recently. You can find it here: https://youtu.be/OWF6YlAAY54?si=AhLY9p9QfLx0puQe

I've heard there might be changes coming to the git integration and deployment process in the future, but no idea when.

Once you've found a suitable answer, be sure to select your answer so other community members can learn from it.