Using Airflow and DBT Core in Fabric

[sreedharshan_10]

We have data in the raw layer and want to transform it using DBT core. Right now we are using DBT on vscode and able work on fabric. When it comes to orchestration I am having some doubts. I am going for the Airflow jobs but the team is suggesting Azure DF. What would be the best and efficient way to orchestrate and if you can find any resources please ping them here 

[NandanHegde · Accepted solution]

You can use notebook to execute your dbt :

https://www.proserveit.com/blog/run-dbt-with-dbt-fabric-adapter-in-fabric-notebook

[sreedharshan_10]

Wow this helps! thank you

[sreedharshan_10]

Thanks! Much Helpful!

[nilendraFabric]

Hello @sreedharshan_10 

I am not able to find particular resources , but here is the summary which will help team to decide which tool to use.

 In simple terms , if your current workflow with dbt on VSCode meets your transformation needs and your team has the technical proficiency to manage more complex, code-driven pipelines, Apache Airflow would be an efficient choice. However, if you prefer tight integration with other Azure services and require a less hands-on orchestration tool, then Azure Data Factory might be the better choice.

Feature	Apache Airflow	Azure Data Factory
Flexibility	High (custom DAGs, detailed dependency management)	Moderate (visual design, less granular control)
dbt Integration	Direct integration with dbt using operators and custom tasks	Integrates with Fabric but may require additional configuration
Ease of Use	Requires Python proficiency and setup expertise	Intuitive, low-code interface integrated within Microsoft Fabric
Ecosystem & Extensibility	Broad community support and extensive plugin ecosystem	Seamless Microsoft integration, managed service environment
Monitoring & Debugging	Rich UI for visualizing DAGs and task states	Built-in monitoring, though slightly less customizable

[sreedharshan_10]

@nilendraFabric But if I look from a cost saving point of view, wouldn't it be advisable if I go with the airflow in fabric? But the downsides are :
1) It is a preview feature
2) A bit complex compared to ADF 
Are the downsides much greater than rewards?

[nilendraFabric]

Hello @sreedharshan_10 

lets see pricing 

• Airflow jobs are charged based on pool uptime and the number of nodes used.

• Two pool types:

  • Starter Pool: Zero-latency startup, shuts down after 20 minutes of inactivity.

  • Custom Pool: Always running for production use.

• Base CU (Consumption Unit) rates:

  • Small: 5 CUs per job.

  • Large: 10 CUs per job.

• Additional nodes:

  • Small: 0.6 CUs per node.

  • Large: 1.3 CUs per node.

• For a single job using a large pool with no extra nodes for one hour:

• Assuming a CU rate of $0.18/CU:
• Cost=10CUs hour×CU rate =$1.80/hour Cost

For a pipeline with one copy activity running for one hour and five orchestration activities:

• Pipeline orchestration: $0.005/hour per activity.

• Data movement: $0.25/hour for cloud-based integration runtime.Total Cost=(1×$0.25)+(5×$0.005)=$0.275Total Cost=(1×$0.25)+(5×$0.005)=$0.275

If your workflows involve complex orchestration and your team is technically skilled, Apache Airflow in Microsoft Fabric offers flexibility but comes at a higher cost for production use due to pool uptime charges.

Let me know for any other questions , happy to discuss