Forum Discussion
Using Airflow and DBT Core in Fabric
We have data in the raw layer and want to transform it using DBT core. Right now we are using DBT on vscode and able work on fabric. When it comes to orchestration I am having some doubts. I am going for the Airflow jobs but the team is suggesting Azure DF. What would be the best and efficient way to orchestrate and if you can find any resources please ping them here
You can use notebook to execute your dbt :
https://www.proserveit.com/blog/run-dbt-with-dbt-fabric-adapter-in-fabric-notebook
6 Replies
Hello sreedharshan_10
I am not able to find particular resources , but here is the summary which will help team to decide which tool to use.
In simple terms , if your current workflow with dbt on VSCode meets your transformation needs and your team has the technical proficiency to manage more complex, code-driven pipelines, Apache Airflow would be an efficient choice. However, if you prefer tight integration with other Azure services and require a less hands-on orchestration tool, then Azure Data Factory might be the better choice.Feature Apache Airflow Azure Data Factory Flexibility High (custom DAGs, detailed dependency management) Moderate (visual design, less granular control) dbt Integration Direct integration with dbt using operators and custom tasks Integrates with Fabric but may require additional configuration Ease of Use Requires Python proficiency and setup expertise Intuitive, low-code interface integrated within Microsoft Fabric Ecosystem & Extensibility Broad community support and extensive plugin ecosystem Seamless Microsoft integration, managed service environment Monitoring & Debugging Rich UI for visualizing DAGs and task states Built-in monitoring, though slightly less customizable nilendraFabric But if I look from a cost saving point of view, wouldn't it be advisable if I go with the airflow in fabric? But the downsides are :
1) It is a preview feature
2) A bit complex compared to ADF
Are the downsides much greater than rewards?Hello sreedharshan_10
lets see pricingAirflow jobs are charged based on pool uptime and the number of nodes used.
Two pool types:
Starter Pool: Zero-latency startup, shuts down after 20 minutes of inactivity.
Custom Pool: Always running for production use.
Base CU (Consumption Unit) rates:
Small: 5 CUs per job.
Large: 10 CUs per job.
Additional nodes:
Small: 0.6 CUs per node.
Large: 1.3 CUs per node.
For a single job using a large pool with no extra nodes for one hour:
- Assuming a CU rate of $0.18/CU:
Cost=10CUs hour×CU rate =$1.80/hour Cost
For a pipeline with one copy activity running for one hour and five orchestration activities:
Pipeline orchestration: $0.005/hour per activity.
Data movement: $0.25/hour for cloud-based integration runtime.Total Cost=(1×$0.25)+(5×$0.005)=$0.275Total Cost=(1×$0.25)+(5×$0.005)=$0.275
If your workflows involve complex orchestration and your team is technically skilled, Apache Airflow in Microsoft Fabric offers flexibility but comes at a higher cost for production use due to pool uptime charges.
Let me know for any other questions , happy to discuss
Thanks! Much Helpful!
You can use notebook to execute your dbt :
https://www.proserveit.com/blog/run-dbt-with-dbt-fabric-adapter-in-fabric-notebook
Wow this helps! thank you
