Forum Discussion

jk-dbt's avatar
jk-dbt
New Member
2 years ago
Solved

DBT Orchestration

Hi, I have a question on dbt orchestration.   I'm not sure how I can trigger a production dbt run in a Fabric pipeline and hoping someone in the community had a suggestion. The dbt-fabric adaptor...
Data Engineering
  • AndyDDC's avatar
    AndyDDC
    2 years ago

    OK thanks.  At the moment you need to run dbt core (and therefore dbt run) on some non-fabric compute unfortunately, like a VM.  

jk-dbt's avatar
jk-dbt
New Member
2 years ago

Hey AndyDDC .  Appreciate you coming back on this.  I'm developing on my local machine.

The basic workflow I'd like to have if possible would be like: 

  • Analytics engineers develop on their local machine.
  • Push to github.
  • scheduled Fabric pipeline somehow clones the repo.
  • Fabric somehow does a dbt run
  • Alert messages get emailed ….

One idea I’ve seen floating around is to have a notebook do the lat 3 steps above.  I actually pip installed dbt in the notebook, got all excited but then got stuck trying to clone the repo.

AndyDDC's avatar
AndyDDC
Most Valuable Professional
2 years ago

OK thanks.  At the moment you need to run dbt core (and therefore dbt run) on some non-fabric compute unfortunately, like a VM.  

  • Jrampono's avatar
    Jrampono
    Regular Visitor
    2 years ago

    This is a great question and one I started to ponder myself about a year ago when I was working at Microsoft.

     

    In my case I wanted to use native scheduling functionality in Fabric and avoid an external Python environment to host dbt. 

     

    It led me to developing a new dbt-adapter that allows the dbt project to be built locally without connection to the fabric workspace and for it to generate notebooks instead of directly execute the sql statements against the Fabric endpoint.

     

     I wanted to use the lakehouse rather than the warehouse in fabric and I wanted to keep all or the transformation logic in spark sql rather than tsql.


    It’s working very well so far and it gives the best of both worlds in my opinion. All of dbt’s benefits such as catalog, lineage, model dependency management, tests etc but with native Fabric artefacts generated. This also allows you to use native fabric and git functionality to migrate through dev, test, prod without having to setup a devops runner to host dbt. I am currently open sourcing the adapter so If you’d like more info hit me up on linked in

     

     https://www.linkedin.com/in/john-rampono-8a73729?utm_source=share&utm_campaign=share_via&utm_content=profile&utm_medium=ios_app