Airflow Scheduler not working when installing Astronomer Cosmos and dbt Fabric

[saceroz]

I’ve tried to replicate the example from this tutorial:
https://learn.microsoft.com/en-us/fabric/data-factory/apache-airflow-jobs-dbt-fabric

However, after adding the following packages to the Apache Airflow requirements:
astronomer-cosmos==1.10.1
dbt-fabric==1.9.5

the Airflow UI shows the following message:

The scheduler does not appear to be running.
The DAGs list may not update, and new tasks will not be scheduled.

Additionally, the Airflow Scheduler status shows as unhealthy.

To restore the Scheduler, I have to remove both packages from the requirements and install only dbt-fabric==1.9.5.
It seems there is some kind of conflict between astronomer-cosmos and Airflow.

Has anyone else encountered this issue, or does anyone have suggestions on how to resolve it?

Thanks

[v-dineshya · Accepted solution]

Hi @saceroz ,

Thank you for reaching out to the Microsoft Community Forum.

The Airflow Scheduler becoming unhealthy when using both astronomer-cosmos==1.10.1 and dbt-fabric==1.9.5 related to dependency conflicts or misconfigurations introduced by astronomer-cosmos, especially in how it interacts with dbt and OpenLineage.

Please try below things to fix the issue.

1. Add target-path to dbt_project.yml, Even though it's deprecated, adding a target-path entry might resolve the OpenLineage-related failures. Please refer below sample YAML code.

target-path: "target"

2. If you are not using OpenLineage, you can disable dataset emission in Cosmos operators. Please refer sample Python syntax.

DbtRunOperator(
task_id="run_dbt",
emit_datasets=False,
...)

3. Please check for Dependency Conflicts, Enter the scheduler container using below sample shell script.

astro dev bash --scheduler

Then manually install packages and check for errors. Please refer below sample shell script.

pip install astronomer-cosmos==1.10.1 dbt-fabric==1.9.5

4. Please try to downgrade astronomer-cosmos to a version as 1.0.5, as newer versions may have introduced breaking changes.

5. Please look into the scheduler logs (logs/scheduler/latest/) for any Python exceptions or import errors that could indicate the root cause.

I hope this information helps. Please do let us know if you have any further queries.

Regards,

Dinesh

[v-dineshya]

Hi @saceroz ,

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. And, if you have any further query do let us know.

Regards,

Dinesh

[v-dineshya]

Hi @saceroz ,

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. And, if you have any further query do let us know.

Regards,

Dinesh