Forum Discussion
Using Airflow and DBT Core in Fabric
You can use notebook to execute your dbt :
https://www.proserveit.com/blog/run-dbt-with-dbt-fabric-adapter-in-fabric-notebook
Hello sreedharshan_10
I am not able to find particular resources , but here is the summary which will help team to decide which tool to use.
In simple terms , if your current workflow with dbt on VSCode meets your transformation needs and your team has the technical proficiency to manage more complex, code-driven pipelines, Apache Airflow would be an efficient choice. However, if you prefer tight integration with other Azure services and require a less hands-on orchestration tool, then Azure Data Factory might be the better choice.
| Feature | Apache Airflow | Azure Data Factory |
| Flexibility | High (custom DAGs, detailed dependency management) | Moderate (visual design, less granular control) |
| dbt Integration | Direct integration with dbt using operators and custom tasks | Integrates with Fabric but may require additional configuration |
| Ease of Use | Requires Python proficiency and setup expertise | Intuitive, low-code interface integrated within Microsoft Fabric |
| Ecosystem & Extensibility | Broad community support and extensive plugin ecosystem | Seamless Microsoft integration, managed service environment |
| Monitoring & Debugging | Rich UI for visualizing DAGs and task states | Built-in monitoring, though slightly less customizable |
nilendraFabric But if I look from a cost saving point of view, wouldn't it be advisable if I go with the airflow in fabric? But the downsides are :
1) It is a preview feature
2) A bit complex compared to ADF
Are the downsides much greater than rewards?Hello sreedharshan_10
lets see pricingAirflow jobs are charged based on pool uptime and the number of nodes used.
Two pool types:
Starter Pool: Zero-latency startup, shuts down after 20 minutes of inactivity.
Custom Pool: Always running for production use.
Base CU (Consumption Unit) rates:
Small: 5 CUs per job.
Large: 10 CUs per job.
Additional nodes:
Small: 0.6 CUs per node.
Large: 1.3 CUs per node.
For a single job using a large pool with no extra nodes for one hour:
- Assuming a CU rate of $0.18/CU:
Cost=10CUs hour×CU rate =$1.80/hour Cost
For a pipeline with one copy activity running for one hour and five orchestration activities:
Pipeline orchestration: $0.005/hour per activity.
Data movement: $0.25/hour for cloud-based integration runtime.Total Cost=(1×$0.25)+(5×$0.005)=$0.275Total Cost=(1×$0.25)+(5×$0.005)=$0.275
If your workflows involve complex orchestration and your team is technically skilled, Apache Airflow in Microsoft Fabric offers flexibility but comes at a higher cost for production use due to pool uptime charges.
Let me know for any other questions , happy to discuss
