Forum Discussion
Warehouse, git integration and dbt
If the Fabric/Git integration does not support .gitignore, I'd expect that no, there is no way to prevent Fabric from tracking certain changes in the Warehouse. The integration looks at the warehouse as one singular object, so picking and choosing what is/is not tracked or deployed hasn't been a feature.
As for workarounds, I can think of two directions you could go depending on where you want to manage your deployment process:
- Somewhere between your dev's branches and your trunk you'd have to create an approach that either cherry picks what you want to merge to trunk, or creates an intermediate branch that can then be modified before changes are merged into trunk.
- If you're looking to source control dbt SQL files within your warehouse, you can use a Schema Compare in visual studio to compare between two warehouses and create deployment scripts. You'd then keep your dev/feature branches separate from your trunk branch, and your commit to the trunk branch would be from your Prod workspace rather than a merge from a feature branch.
- If you want to see what the schema compare looks like, I made a video on my Youtube channel on this approach recently. You can find it here: https://youtu.be/OWF6YlAAY54?si=AhLY9p9QfLx0puQe
I've heard there might be changes coming to the git integration and deployment process in the future, but no idea when.
Once you've found a suitable answer, be sure to select your answer so other community members can learn from it.
Hi Anonymous ,
Thanks for the reply from AndyDDC / DataBard .
Currently, Fabric does not provide a way to prevent tracking of Data Warehouse files in Git integrations.
You can manually delete unwanted files before committing to Git.
I understand that this can be cumbersome, but it's a workaround until there's a better solution.
If you have a better solution please share it here, it will be of great help to users who are experiencing similar issues.
Best Regards,
Yang
Community Support TeamIf there is any post helps, then please consider Accept it as the solution to help the other members find it more quickly.
If I misunderstand your needs or you still have problems on it, please feel free to let us know. Thanks a lot!Sorry for slow response.
We are using dbt core a cli tool that is not dependant on dbt cloud: https://docs.getdbt.com/docs/core/installation-overview
Since gitignore is not supported and we want to have the warehosue in the same workspace as the other fabric items, we have opted for the following:
- We have created a script that checks if any changes are made to the warehouse other than creation of one. This script is ran when a pulle request is opened and needs to succeed before merge. This prevents unwanted changes to be commited to main.
- Direct commits to main is disabled, meaning changes made to the Warehouse in production via dbt will not be comitted.
Thanks again, everyone!
Thanks all!
- Good idea. Yes, that will work. At least the warehouse won't be tracked in Git. Ideally we would like it all to stay in the same workspace, as it all belong to the same data platform.
Is dbt running within a Notebook? I assume that's how it's version controlled?
- The dbt project is checked into git (same repo as the fabric code) just like the jaffle shop example from dbt.
- We run dbt build from an Azure Devops Pipeline, which sends of the sql statements to the Warehouse. So the orchestration it self is not currently handled inside Fabric.
On that note, I see that dbt cli support for data factory is scheduled for Q1 2025. Given you work with Warehouse and that is the only (OLAP) service dbt is compatible with in Fabric, do you know anything about this upcomming feature? https://learn.microsoft.com/en-us/fabric/release-plan/data-factory#data-pipeline-support-dbt
Anonymous:
Thanks for clarification. The deletion of files or at least prevent these files from being merged to main is something we can automate with Azure Devops Pipelines. We will test this one out, thanks!
Thanks DataBard. The first option you proposed is something i will test out. I'll also have a look at your youtube video, thanks!
Ah ok. So you're running Dbt cloud?
I don't know much about the Dbt integration in data factory other than someone in MS accidentally switched it on a few months ago but although there was a Dbt task, it didn't actually work.
hopefully we'll get some more info soon
Sorry for slow response.
We are using dbt core a cli tool that is not dependant on dbt cloud: https://docs.getdbt.com/docs/core/installation-overview
Since gitignore is not supported and we want to have the warehosue in the same workspace as the other fabric items, we have opted for the following:
- We have created a script that checks if any changes are made to the warehouse other than creation of one. This script is ran when a pulle request is opened and needs to succeed before merge. This prevents unwanted changes to be commited to main.
- Direct commits to main is disabled, meaning changes made to the Warehouse in production via dbt will not be comitted.
Thanks again, everyone!