Apache Airflow Job fails to run dbt project connected to Azure DevOps

[krisjones]

I have a dbt project with a DAG to run it. This project is stored on Azure DevOps. 

When I use File Storage --> Fabric managed i.e. I author the code in the UI, my code runs fine and completes.  

When I connect Apache Airflow Job to my Azure DevOps repo with the EXACT SAME CODE, I get the error in the attached screenshot. 

I have been through all the dbt/Airflow links from Microsoft. I have configured the dbt project exactly the same.

I have confirmed I'm using the correct version of dbt-fabric.

I don't have anything for the requirements. 

I know the Apache Airflow Job is in preview state.

No matter what I try, I get the "Command exited with return code 2" error. 

How can I solve this issue. 

[krisjones]

Running dbt debug gives me the same error code of 2.

But what I did is, I removed the dbt-fabric 1.5.0 from the requirements and I got the dbt: command not found error. So dbt is being installed. 

I have Airflow with dbt running in a container locally and there everything works. Also simply running dbt from the command line works also. So profiles/credentials are all good. 

It must be something else. 

[v-tsaipranay]

Hi  , 

Thanks for the details on your Apache Airflow job issue with the dbt project in Microsoft Fabric. After fixing the initial dbt: command not found error by adding dbt-fabric==1.5.0, dbt debug now fails with exit code 2. Since your local setup is fine, the problem seems to be with the Fabric Airflow environment.

Please try this troubleshooting steps to resolve the issue:

• Use the FabricItemOperator (recommended for Fabric items) instead of BashOperator. Configure it with your dbt project’s item_id and workspace_id from the Fabric UI. Example:
  
  

  from airflow.providers.microsoft.fabric.operators.fabric import FabricItemOperator

  from airflow import DAG

  from datetime import datetime

  with DAG(dag_id="dbt_fabric_dag", start_date=datetime(2025, 3, 30), schedule_interval=None) as dag:

      dbt_task = FabricItemOperator(

          task_id="run_dbt",

          item_type="dbtProject",

          item_id="your-dbt-project-id",

          workspace_id="your-workspace-id",

      )

•  Ensure “Apache Airflow requirements” includes dbt-core==1.5.0 and dbt-fabric==1.5.0. Check compatibility with the Fabric Python version.

• Run dbt debug --log-level=debug --profiles-dir ~/.dbt > /tmp/dbt_debug.log 2>&1 via BashOperator if needed and review the log for specific errors.
• If using BashOperator, set the PATH: env={"PATH": "/path/to/venv/bin:$PATH"} and activate the virtual environment.
• If using BashOperator, set the PATH: env={"PATH": "/path/to/venv/bin:$PATH"} and activate the virtual environment.

I am also including documentation as a reference for your better understanding:

Apache Airflow and dbt in Microsoft Fabric | by Christopher Nagl | Feb, 2025 | Medium

Run a Fabric data pipeline and notebook using Apache Airflow DAG. - Microsoft Fabric | Microsoft Lea...

If the issue persists, consider raising a support ticket with Microsoft Fabric for further investigation. Please use the following link as a guide for raising the support ticket:

How to create a Fabric and Power BI Support ticket - Power BI | Microsoft Learn

If this post helps, then please give us Kudos and consider Accept it as a solution to help the other members find it more quickly.

Thankyou.

[krisjones]

Thank you for your response but everything you are suggesting I have done already, read or looked up. I will raise a support ticket with Microsoft. 

[v-tsaipranay]

Hi @krisjones ,

Could you please confirm if the issue has been resolved after raising a support case? If a solution has been found, it would be greatly appreciated if you could share your insights with the community. This would be helpful for other members who may encounter similar issues.

Thank you for your understanding and assistance.

[krisjones · Accepted solution]

I already wrote about this here: Solved: Re: Issues resuming Apache Airflow Job in Fabric - Microsoft Fabric Community

I didn't raise a support ticket because Airflow consumes too many resources vs the capacity that we bought. I decided to containerize the dbt project instead and call it in a Web Activity using REST API.

[v-tsaipranay]

Hello @krisjones ,

Appreciate for your time and for sharing your experience with us! Since you have found a workaround, please mark your answer as a solution to help others identify it easily.

Thankyou.

[v-tsaipranay]

Hi @krisjones ,

Thank you for posting in the Microsoft Fabric Community. Also thankyou @Vinodh247  for providing inputs.

To fix your Apache Airflow Job failing with the "return code 2" error when running your dbt project from Azure DevOps, the issue likely stems from differences between the Fabric UI and Airflow environments, such as missing configurations or permissions.

First, update your Airflow DAG to capture detailed logs by running dbt debug and dbt run with verbose output, saving the results to a file for review.

Next, ensure the dbt run command is executed from the correct directory in your cloned Azure DevOps repository, and verify that your profiles.yml file is properly set up for the dbt-fabric adapter with the right Fabric workspace and authentication details.

Check that Airflow has permission to access the Azure DevOps repository by testing a manual git clone in your DAG and confirm that the dbt-fabric package is installed in the Airflow environment through the Fabric UI.

 If the issue persists, test the dbt command locally to isolate the problem, and consider reaching out to Microsoft Support with your logs, as Airflow Jobs in Fabric are in preview and might have limitations.

I hope this will resolve your issue, if you need any further assistance, feel free to reach out.

If this post helps, then please give us Kudos and consider Accept it as a solution to help the other members find it more quickly.

Thankyou.

[Vinodh247]

A non‐zero exit code in dbt usually means dbt can’t initialize or find what it needs (credentials, the correct python packages??). though the code is the same as what you use in the fabric ui, your airflow environment most likely differs in minor ways.

The most common cause may be that your fabric environment is “auto‐provisioned” in the UI but not in Airflow. Double‐check that dbt-fabric is installed and that your profiles/credentials for Fabric are correctly set in Airflow. Verifying via dbt debug in the same container/virtual environment where Airflow runs is usually the quickest way to narrow down what is missing.