Metadata Driven DAGs in Airflow

Hi tayloramy ,

Thanks for reaching out to the Microsoft Fabric Community.

At this time, Microsoft Fabric does not include native support for fully metadata-driven DAGs within its Apache Airflow integration. Airflow in Fabric uses standard Python-based DAG definitions, and while you can parameterize them using variables or external configuration, a fully metadata-driven orchestration approach requires custom implementation.

 

You may want to review this open-source project: Metadata-Driven ETL Framework for Complex Workflows in Apache Airflow . It demonstrates how metadata tables (stored in SQL Server or Postgres) can define job configurations, task sequences, and ETL logic, with Airflow reading and executing based on that metadata. This design pattern could be adapted in Fabric by sourcing metadata from a Lakehouse or Data Warehouse.

 

Some useful docs for reference:

What is Apache Airflow job? - Microsoft Fabric | Microsoft Learn

Dynamic Dag Generation — Airflow 3.1.0 Documentation

 

Hope this helps. Please reach out for further assistance.

Thank you.