Forum Discussion

Anonymous's avatar
Anonymous
Not applicable
1 year ago
Solved

Warehouse, git integration and dbt

Hi all,   TL;DR: Is it possible to disable tracking of changes in data warehouse files in Fabric’s Git integration? If so, how? We have tried .gitignore and Fabric does not hounor that.    Contex...
data warehouse
warehouse
Anonymous's avatar
Anonymous
Not applicable
1 year ago

Thanks all! 

 

AndyDDC:

- Good idea. Yes, that will work. At least the warehouse won't be tracked in Git. Ideally we would like it all to stay in the same workspace, as it all belong to the same data platform. 

 

Is dbt running within a Notebook?  I assume that's how it's version controlled?

- The dbt project is checked into git (same repo as the fabric code) just like the jaffle shop example from dbt. 

- We run dbt build from an Azure Devops Pipeline, which sends of the sql statements to the Warehouse. So the orchestration it self is not currently handled inside Fabric.  

 

On that note, I see that dbt cli support for data factory is scheduled for Q1 2025. Given you work with Warehouse and that is the only (OLAP) service dbt is compatible with in Fabric, do you know anything about this upcomming feature? https://learn.microsoft.com/en-us/fabric/release-plan/data-factory#data-pipeline-support-dbt 

 

Anonymous:

Thanks for clarification. The deletion of files or at least prevent these files from being merged to main is something we can automate with Azure Devops Pipelines. We will test this one out, thanks!

 

Thanks DataBard. The first option you proposed is something i will test out. I'll also have a look at your youtube video, thanks!

 

 

 

 

AndyDDC's avatar
AndyDDC
Icon for Most Valuable Professional rankMost Valuable Professional
1 year ago

Ah ok. So you're running Dbt cloud?

I don't know much about the Dbt integration in data factory other than someone in MS accidentally switched it on a few months ago but although there was a Dbt task, it didn't actually work.

hopefully we'll get some more info soon

  • Anonymous's avatar
    Anonymous
    Not applicable
    1 year ago

    Sorry for slow response. 

     

    We are using dbt core a cli tool that is not dependant on dbt cloud: https://docs.getdbt.com/docs/core/installation-overview

     

    Since gitignore is not supported and we want to have the warehosue in the same workspace as the other fabric items, we have opted for the following: 

     

    - We have created a script that checks if any changes are made to the warehouse other than creation of one. This script is ran when a pulle request is opened and needs to succeed before merge. This prevents unwanted changes to be commited to main. 

    - Direct commits to main is disabled, meaning changes made to the Warehouse in production via dbt will not be comitted. 

     

    Thanks again, everyone!