Forum Discussion
stuck with MS Learn Tutorial for Apache Airflow Jobs for dbt-fabric Orchestration
Hi rabbyn
As discussed in the support ticket the workaorund solution is to chnaged the below configuration to resolve the issue.- astronomer-cosmos==1.10.1
- dbt-fabric==1.9.5 Updated the right requirements for Apache Airflow.
However, if you're still facing challenges, feel free to share the details, and we'll be happy to assist you further.
Looking forward to your response!
Best Regards,
Community Support Team _ C Srikanth.
Community SupportHi rabbyn
Thanks for reaching out to Fabric Community.
Here are the few check points that might resolve your issue.
- Ensure the required environment variables (FABRIC_WORKSPACE_ID, FABRIC_WAREHOUSE_ID, FABRIC_CAPACITY_ID) are explicitly defined in the "Environment variables" section of the Airflow configuration in Fabric.
- Even if the requirements.txt file validates successfully with astronomer-cosmos==1.5.1 and dbt-fabric==1.5.0, re-deploy the environment from scratch to eliminate issues caused by corrupted or cached dependencies.
- Check the Airflow logs for any Python-related import errors such as No module named cosmos or dbt not found, which indicate module loading issues during DAG startup.
- Verify that your my_cosmos_dag.py file is located directly under the /dags directory and follows correct DAG declaration syntax required by Airflow.
- The Starter Pool in Fabric may impose limits on compute or task parallelism, so reduce task concurrency if your DAG is calling resource-heavy operations like dbt transformations.
Suggested troubleshooting steps:
Add your .env or DAG-level configuration using:
import os
os.environ["FABRIC_WORKSPACE_ID"] = "<your-workspace-id>"
os.environ["FABRIC_CAPACITY_ID"] = "<your-capacity-id>"
os.environ["FABRIC_WAREHOUSE_ID"] = "<your-warehouse-id>"
If you are using DbtDag, validate all dbt_kwargs and project paths:
DbtDag(
project_dir="/usr/local/airflow/dags/dbt",
profiles_dir="/usr/local/airflow/dags/dbt",
# other parameters
)
Rebuild the environment or delete and recreate the Airflow workspace to clear possible internal caching.
If the above information helps you, please give us a Kudos and marked the Accept as a solution.
Best Regards,
Community Support Team _ C Srikanth.
Hi v-csrikanth , thanks for your support. I did follow all the troubleshooting steps you mentioned but no success, despite the requirements.txt being "validated" the Airflow cluster is not starting up (time out after 18min). I tried with starter cluster then I recreate a brand new Apache Airflow from an other Workspace and the outcome is the same. Below the screenshot you find a code snippet from my latest DAG script (based on your instructions , I something is wrong let me know). Thanks
import os from pathlib import Path from datetime import datetime from cosmos import DbtDag, ProjectConfig, ProfileConfig, ExecutionConfig # Add environment variables here os.environ["FABRIC_WORKSPACE_ID"] = "18584879-394b-****-8d34-61e864c0bd1c" os.environ["FABRIC_CAPACITY_ID"] = "FAC80AA7-5E69-****-88E7-DE388FC23422" os.environ["FABRIC_WAREHOUSE_ID"] = "60a45649-7e92-****-90b3-237243a35114" DEFAULT_DBT_ROOT_PATH = Path(__file__).parent.parent / "dags" / "nyc_taxi_green" DBT_ROOT_PATH = Path(os.getenv("DBT_ROOT_PATH", DEFAULT_DBT_ROOT_PATH)) profile_config = ProfileConfig( profile_name="nyc_taxi_green", target_name="fabric-dev", profiles_yml_filepath=DBT_ROOT_PATH / "profiles.yml", ) dbt_fabric_dag = DbtDag( project_config=ProjectConfig( project_dir="/usr/local/airflow/dags/dbt", profiles_dir="/usr/local/airflow/dags/dbt", ), operator_args={"install_deps": True}, profile_config=profile_config, schedule_interval="@daily", start_date=datetime(2024, 9, 10), catchup=False, dag_id="dbt_fabric_dag", )
