Airflow in Fabric - Have you made this work for your organisation?

[JamesJ5]

Hello,

Our team has been working on a strategic assessment. We wanted to see how Fabric could transform our current analytics and data operations going forward. We have taken a reporting business use case initially to verify the approach before full adoption. 

The team chose to work with DBT and Airflow and we set up IaC so we could automate pipelines to ingest from a single source as well as pause/resume FCU (F8) when outside office hours. We wanted to demonstrate an automated pipeline so set up Fabric to work with git sync. Initially we used an Airflow Pipeline with a starter pool, which worked well and did as it says and stopped consuming compute after 20 minutes of inactivitiy. This fit our use case. Our DBT project repo is about 3MB and we need to run it once per day for about 30 minutes to sync the latest database updates.

However, we found that sometimes Airflow wasn't running, which would completely fail the DBT run. So we set up a custom pool and added 3 nodes. Not only would this sometimes not be available (in both cases having to manually start it up again) we found from the monitors that Airflow would consume all the FCU often pushing us over 100% FCU usage even when nothing was happening in Fabric. Additionally we experienced an error with git-sync that just said 'Bad request' where Airflow would not sync the repo (3MB). The solution to this was to add 2 more nodes and then git sync starts working again. Now though, our compute is entirely consumed by Airflow. So for a starter use case syncing up a relatively small amount of business data to use for business analytics/insights costs the princely sum of circa $2000/month. Not the sustainability or scalability story we want to promote to executives, who would most likely push for the continued use of manual labour and spreadsheets with this kind of data.

We are a mid-sized enterprise looking for meaningful ways to transform data capability and culture. I expect there are likely many businesses tackling this same set of circumstances. We are certainly within the target market that Microsoft is promoting this platform to. I wonder, therefore have others been here and tackled this entry into the world of using Fabric for your organisation and starting small to demonstrate value. Have you encountered these issues? If so, how have you solved them? If so, I would be very interested (and appreciative) of your thoughts and approach. 

I think Microsoft has a great product here, and the move to include Airflow and DBT within Fabric was and is brilliant. However, the barriers mentioned above seriously dent its credibility as a practical and cost-efficient solution. Fabric platform has a lot to offer and I hope it succeeds. Unfortunately though to prove a strategic adoption, developers need tools they know and trust to work well (and data devs do love DBT and Airflow) and the business needs trustworthy automation and scalability. My team have come up with ideas how to solve for this, but they move outside Fabric and that erodes the platform experience from the business perspective (we are a Microsoft organisation so Fabric would fit in perfectly).

I look forward to hearing your ideas on this matter.

Thank you

James

[v-prasare]

We are following up once again regarding your query. Could you please confirm ifyou have raised ideas with Microsoft?
If not please raise idea and upvote it. If we don’t hear back, we’ll go ahead and close this thread.
Should you need further assistance in the future, we encourage you to reach out via the Microsoft Fabric Community Forum and create a new thread. We’ll be happy to help.

[Shahid12523]

Airflow inside Fabric works but is resource-heavy: it keeps consuming FCUs even when idle, git-sync can be flaky, and small dbt jobs can cost thousands/month.

Run Airflow outside Fabric (AKS, ADF, MWAA) and trigger Fabric jobs → pay only for actual compute.

Or replace Airflow with Fabric Pipelines for native orchestration.

For dbt, run in Notebooks or Pipelines directly.

Use starter pools for PoC to keep costs down.

👉 Bottom line: today, Airflow in Fabric isn’t cost-efficient for small workloads; hybrid or native Fabric pipelines are more practical.

[JamesJ5]

You are absolutely right. The solution in theory is good, but in practice is not easy to use. Given that these tools are common for data devs and that Microsoft has made the decision to build them into the Fabric experience, they need to iron these issues out. At the moment our solution (as we only run DAGs once a day for about 30mins) is to use a CI run to kickstart the starter pool into action, then run a job on it and then stop the starter pool if it hasn't already and hour later. When it works it works well and limits the cost of using Fabric making it afordable for such small use case. However, we have seen conflicts with packages, error logs that are too opaque to do anything with etc. All points to the backend not functioning well. Also, when fixes from Microsoft have been implemented and worked, they fail again in a few days, suggesting that on the backend stability is an issue. 

I am not sure raising an idea to Microsoft is of use as the advertised functionality is what we want and I have seen it work well and produce the desired business benefits. I think Microsoft needs to ensure that the harness for this feature/tool ensures stability as they release upgrades.

Thank you for your reply.

[v-prasare]

Hi @JamesJ5,

Thank you for providing such a clear and thoughtful overview of your experience with Fabric. your team has put considerable effort into exploring its potential, and we truly value the feedback. Your observations shed light on both the opportunities and the challenges of adopting the platform in real-world scenarios, and we fully recognize how these issues can affect confidence among both developers and business leaders.

I’d encourage you to submit your detailed feedback and ideas via Microsoft's official feedback channels, such as the Microsoft Fabric Ideas. Feedback submitted here is often reviewed by the product teams and can lead ...

Thanks,

prashanth

MS Fabric community support