topic Re: dbt CLI support, what can Fabricators expect from the upcoming release? in Pipelines

dbt CLI support, what can Fabricators expect from the upcoming release?

stefanberreiter — Wed, 12 Jun 2024 12:09:44 GMT

Hi Fabric Community,

we are using dbt core since a while in conjunction with MS Fabric.

I am wondering if you can share what we can "roughly" expect with the dbt CLI data pipeline support (see link) you plan to release in Q3. [1]

a) Is it aimed at dbt cloud or dbt core?

b) If dbt core is supported, do you plan to add a way to specify e.g. the main branch of a Azure DevOps repo and then a sequence of dbt CLI commands like Databricks does? [2]

Happy to share any information you have, as it has some implications on our analytics roadmap (as we'd love to use dbt core instead of stored procedures).

If there's a way to get access to a private preview, let me know.

Greetings from Austria
Best, Stefan

[1] What's new and planned for Data Factory in Microsoft Fabric - Microsoft Fabric | Microsoft Learn

[2] Use dbt transformations in a Databricks job | Databricks on AWS

Re: dbt CLI support, what can Fabricators expect from the upcoming release?

Anonymous — Thu, 13 Jun 2024 08:09:09 GMT

Hi stefanberreiter
Thanks for using Fabric Community.
At this time, we are reaching out to the internal team to get some help on this. We will update you once we hear back from them.
Thanks

Re: dbt CLI support, what can Fabricators expect from the upcoming release?

Anonymous — Fri, 14 Jun 2024 18:13:15 GMT

Hi stefanberreiter
This is designed for dbt core scenarios. If you’re currently using dbt core, this service is intended for you. It allows you to execute dbt commands to run your dbt models. We will provide a managed dbt command-line interface (CLI) for each run in a serverless fashion, and you will be billed only for the duration of the dbt activity run. (Supported adaptors in the first milestone: Fabric DW, Fabric LH/ Spark, Snowflake.)
Initially, we will support reading your dbt files from a OneLake (LH files), but we plan to extend this support to GitHub in the near future.

Hope this helps.

Re: dbt CLI support, what can Fabricators expect from the upcoming release?

Anonymous — Mon, 17 Jun 2024 15:13:52 GMT

Hi stefanberreiter
We haven’t heard from you on the last response and was just checking back to see if your query got resolved. Otherwise, will respond back with the more details and we will try to help.
Thanks

Re: dbt CLI support, what can Fabricators expect from the upcoming release?

stefanberreiter — Mon, 17 Jun 2024 19:28:20 GMT

Hi,
thanks for providing an answer to my question.
Reading the dbt files from the LH files area sounds "interesting". Once the dbt project is hosted in some Git Repo (let's say ADO), this would mean pulling the repo to the LH Files area?
Or "deploying" via uploading the dbt project as folder? How would the deployment experience look like?

One wish for us early users would then be a pragmatic how-to tutorial how to do it.
I'm wondering why you are going for GitHub in the first iteration and not ADO, as the latter seems to be the go-to VCS for Fabric?

In any case, thanks for your answers so far.