Forum Discussion
Using Airflow and DBT Core in Fabric
You can use notebook to execute your dbt :
https://www.proserveit.com/blog/run-dbt-with-dbt-fabric-adapter-in-fabric-notebook
nilendraFabric But if I look from a cost saving point of view, wouldn't it be advisable if I go with the airflow in fabric? But the downsides are :
1) It is a preview feature
2) A bit complex compared to ADF
Are the downsides much greater than rewards?
Hello sreedharshan_10
lets see pricing
Airflow jobs are charged based on pool uptime and the number of nodes used.
Two pool types:
Starter Pool: Zero-latency startup, shuts down after 20 minutes of inactivity.
Custom Pool: Always running for production use.
Base CU (Consumption Unit) rates:
Small: 5 CUs per job.
Large: 10 CUs per job.
Additional nodes:
Small: 0.6 CUs per node.
Large: 1.3 CUs per node.
For a single job using a large pool with no extra nodes for one hour:
- Assuming a CU rate of $0.18/CU:
Cost=10CUs hour×CU rate =$1.80/hour Cost
For a pipeline with one copy activity running for one hour and five orchestration activities:
Pipeline orchestration: $0.005/hour per activity.
Data movement: $0.25/hour for cloud-based integration runtime.Total Cost=(1×$0.25)+(5×$0.005)=$0.275Total Cost=(1×$0.25)+(5×$0.005)=$0.275
If your workflows involve complex orchestration and your team is technically skilled, Apache Airflow in Microsoft Fabric offers flexibility but comes at a higher cost for production use due to pool uptime charges.
Let me know for any other questions , happy to discuss