Issues resuming Apache Airflow Job in Fabric

[krisjones]

I have Fabric capacity deployed to North Europe.  Apache airflow is available in North Europe. Apache Airflow Job is in preview state. This is what I see in the UI. 

I constantly have issues resuming the Airflow pool. I keep getting an InternalError. Sometimes I get a message saying 'ensure all listed requirements are valid and compatible with the current Apache Airflow version'. The only package I have is

dbt-fabric==1.9.2. The only way around this is to create a new Apache Airflow job, then I don't get this error and I can run some DAGs. I understand the Airflow jobs are in preview state but this to me looks like it's an alpha version and it's not usable. I would expect previews to have some bugs but be at least something that resembles a beta version and you can use.
Is there anything I can do, check, inspect or change to help with this problem?
 

[krisjones]

I'm aware of how to run dbt in airflow in Fabric. I've been through all those links before.

Unfortunately, whether I have something in the requirements file or not, the pool does not start after it has been stopped. 

[v-veshwara-msft]

Thanks for the clarification. Since the issue occurs regardless of whether there’s anything in the requirements.txt file, it seems to be related to the Airflow pool itself rather than package compatibility.

As this feature is still in preview, there may be limitations affecting the ability to resume stopped Airflow pools. While there isn't a confirmed workaround for this at the moment, I’d recommend reporting this through Microsoft support so the engineering team can investigate further.
Link for support ticket: Microsoft Fabric Support and Status | Microsoft Fabric

In the meantime, you might find these related discussions helpful:
Solved: Re: Programmatically pause pool - Microsoft Fabric Community
Re: Pool stuck in "Resuming" status - Microsoft Fabric Community

Pool will not start following unresponsive schedul... - Microsoft Fabric Community

Since Airflow Jobs are still in preview, improvements are likely on the way. We appreciate your patience while this gets addressed. Let us know if you’d like any further assistance.

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly and a kudos would be appreciated.

Regards,
Vinay.

[v-veshwara-msft]

Hi @krisjones ,

Could you confirm if your issue has been resolved after raising a support case? If a solution was found, we’d appreciate it if you could share your insights with the community, as it may help others facing similar issues.

Thank you for your support and cooperation!

[krisjones · Accepted solution]

In the end we decided not to go with Airflow in Fabric because it's consuming too many CU versus the capacity we are paying for, thus I have not raised a ticket.

I decided to containerize dbt project and call it through using the REST API in a Web Activity in a Data Factory Pipeline. This is how we'll run dbt for our transformation jobs:

Run DBT in Azure Data Factory: A Clean Solution | by Guang X | Medium

Data Factory is included in the capacity we are paying for. 

[v-veshwara-msft]

Thank you for sharing your approach. Containerizing your dbt project and triggering it via REST API in Data Factory sounds like a well-optimized solution, especially considering capacity usage. This insight will be valuable to others facing similar challenges.

Since you've found an alternative approach that works for your scenario, would you mind marking your reply as the accepted solution? This will help other community members who come across this thread find a useful resolution more quickly.

Appreciate your time and for sharing your experience with us! Let us know if you need any further assistance.

Thank you.

[v-veshwara-msft]

Hi @krisjones ,

Thanks for sharing your experience in Fabric Community.
Since Apache Airflow Jobs in Fabric are still in preview, it’s expected that there might be some blockers, but repeatedly running into InternalError when resuming the Airflow pool can be disruptive. 
Here are few things that might help.

1. The latest version of dbt-fabric is 1.9.3, released in March. If you're using 1.9.2, try upgrading to 1.9.3 by updating your requirements.txt file. Compatibility issues could be contributing to the problem.

2. Also, make sure that any other dependencies (like astronomer-cosmos) are aligned with your Apache Airflow version.

3. While creating a new Airflow job works as a workaround, but that’s not ideal, it does suggest that the issue could be related to the Airflow pool state. This might get better as Microsoft continues improving the preview feature.

Transform data using dbt - Microsoft Fabric | Microsoft Learn

Since this feature is in preview, there may be improvements on the way. 

Hope this helps. Please reach out for further assistance.

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly and a kudos would be appreciated.

Regards,
Vinay.