topic Transform data using dbt tutorial, not working in Apache Airflow Job

Transform data using dbt tutorial, not working

BiJoe — Thu, 03 Oct 2024 13:59:03 GMT

After successfully starting the Hello Airflow job example, I am trying a slightly modified version of this tutorial, in a new Airflow job:

https://learn.microsoft.com/en-us/fabric/data-factory/apache-airflow-jobs-dbt-fabric

The modification is that I am connecting to the default Warehouse with sample data, instead of the mentioned Contoso warehouse.

The problem is that the DAG does not run, after trying multiple schedule intervals.

Hopefully I have configured the DAG correctly, the only doubt I have is the database field in the profiles YAML file. Should that be the name of the warehouse?

EDIT:

I just discovered that there is a schedule interval set to daily, which means that the DAG will run at midnight on the day it is started, right? Is this a particulary beginner friendly setting for a tutorial? The tutorial also says to "press play", but this is a scheduled DAG right?

EDIT 2: I have set the schedule interval to continuous, and start date two days ago, catchup to False still no DAGs are run. Where can I find the logs?

Screenshots:

Re: Transform data using dbt tutorial, not working

Anonymous — Fri, 04 Oct 2024 02:23:04 GMT

Hi BiJoe

In Fabric, the database field should be the name of the warehouse you want to connect to. Make sure it matches the default warehouse name in the Fabric with the sample data.

If the scheduling interval is set to daily, the DAG will run at midnight (UTC time) on the day it starts.

Even if the DAG is a scheduled task, you can trigger it manually by pressing the Play button in Airflow UI.

About Monitoring DAG:

https://learn.microsoft.com/en-us/fabric/data-factory/apache-airflow-jobs-run-fabric-item-job#monitor-your-dag

Regards,

Nono Chen

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

Re: Transform data using dbt tutorial, not working

BiJoe — Fri, 04 Oct 2024 08:47:30 GMT

Anonymous thanks for replying.

Unfortunately there is no Play button in Airflow UI for the DAG I have replicated from the tutorial.

Thanks for the link about monitoring DAGs, but that does not work so long the DAG does not run properly.

Re: Transform data using dbt tutorial, not working

nsprudhvi — Wed, 23 Oct 2024 12:12:37 GMT

Hi BiJoe,

Yes, you're correct! When a DAG is set with a daily schedule interval, it will run at midnight on the day it is started. While this can be a common setting, it might not be the most beginner-friendly option for a tutorial since it can lead to confusion about immediate execution.

Regarding the "press play" instruction, that typically refers to manually triggering the DAG. You can start a DAG by clicking on the three dots in the Airflow UI and selecting "Trigger DAG."

As for your edited question about finding the logs, you can access the logs within the Airflow UI. Once you trigger a DAG, you can click on the DAG name, then go to the "Graph View" or "Gantt View," and click on the task you want to inspect. This will lead you to the logs for that specific task or click on the logs button inside a DAG in Airflow UI.

If you still encounter issues with your DAGs not running, double-check your configurations and the Airflow scheduler status.

Let me know if you need further assistance!

Best regards,
Sai Prudhvi Neelakantam
Data Engineer | 3x Microsoft Certified

LinkedIn: in/saiprudhvineelakantam | geekindata.com

If this helps, kindly mark this as the accepted solution! 👍

Re: Transform data using dbt tutorial, not working

Anonymous — Wed, 22 Jan 2025 14:58:25 GMT

Hi BiJoe ,

Could you please tell me what was the exact solution? I have the same problem you had and I cannot really understand from the 'solved' reply what was the solution....

Re: Transform data using dbt tutorial, not working

BiJoe — Thu, 23 Jan 2025 13:22:52 GMT

Hi Anonymous I doubt that I set the status so "Solved" myself, not really sure. It was not working properly IIRC. Haven't touched this since back then.