Forum Discussion
Issue with “Always On Pool” for Airflow job causing Git sync error in Microsoft Fabric
Hi everyone,
I’m running into a blocking issue when using the Always On Pool setting for an Airflow job in Microsoft Fabric. Here’s a summary of the scenario and the error:
Scenario:
I enabled the “Always On Pool” for our Airflow job (so it remains running for scheduling).
That change updates the apacheairflowjob-content.json, modifying pool fields from “StarterPool” to “CustomPool/AlwaysOn”.
In our shared “Dev” workspace this works fine (job is deployed via CI/CD).
However, when a developer pulls from Git into their local Fabric workspace, the sync fails with:
Workload Error Code: RequestValidationFailed
Workload Error Message: ‘AirflowFile’ CustomPoolCreationWithAirflowFiles ‘Creating Airflow with CustomPool and Airflow Files in the same operation is unsupported. Please create the Airflow artifact separately first.’
Problem:
The combination of “CustomPool/Always On” + Airflow files triggers a validation error during the Git-based workspace sync. This blocks developers from updating their workspaces.
Current workaround:
We temporarily ask developers to remove the Airflow job item from their incoming changes before merging/pulling. But this is messy because when they open a PR the files appear missing and must be re-added manually.
Questions:
Is this behaviour by design (i.e., unsupported scenario in Fabric)?
Is there a supported way to exclude a specific Airflow job item from Git synchronization within a workspace?
Are there recommended best practices for using Always On pools for Airflow jobs in Fabric and maintaining developer workspace Git sync?
Thanks in advance for any guidance!
Hi everyone,
Just wanted to share an update on this issue in case it helps others facing the same problem.
We’ve implemented a workaround that allows us to keep the “Always On Pool” configuration for the Airflow job while avoiding the Git synchronization error in Fabric.
What we did:
Reverted the apacheairflowjob-content.json file in Git to use the default StarterPool values.
Updated our deployment automation to dynamically replace these values during deployment, switching them to Always-On (CustomPool) using environment variables in the pipeline.
This ensures that developers can now sync their Fabric workspaces normally without validation errors, while the shared environment still runs the Airflow job in Always-On mode.
So far, this workaround has been tested and works correctly.
Hopefully, this helps others until Microsoft provides official support for Always-On pool configurations in Git-synced workspaces.
4 Replies
Hi everyone,
Just wanted to share an update on this issue in case it helps others facing the same problem.
We’ve implemented a workaround that allows us to keep the “Always On Pool” configuration for the Airflow job while avoiding the Git synchronization error in Fabric.
What we did:
Reverted the apacheairflowjob-content.json file in Git to use the default StarterPool values.
Updated our deployment automation to dynamically replace these values during deployment, switching them to Always-On (CustomPool) using environment variables in the pipeline.
This ensures that developers can now sync their Fabric workspaces normally without validation errors, while the shared environment still runs the Airflow job in Always-On mode.
So far, this workaround has been tested and works correctly.
Hopefully, this helps others until Microsoft provides official support for Always-On pool configurations in Git-synced workspaces.
- svenchio
Super User
Hi crisvaillant, let's face a reallity of Fabric ... "what you see is what you get", anything else is basically in process of being built (and that's a big "if" depending on the roadmap) and everyelse, has to be ask and rally people to vote for your idea to be implemented OR and this is the point of this post, BUILD YOURSELF! How? Interacting with the API Microsoft Fabric REST API references - Microsoft Fabric REST APIs | Microsoft Learn or the new CLI for Fabric fabric-cli
In this context, the answers to the question you ask are as follow
#1. Yes, although all items are not supported by Git, and remember that it was until very recent that was announced, most likely some not 100 of the properties are track within these items, causing situatinos like this, and many others yet to be discover, so, back to my original stament of "what you see is what you get" if you have not found a way to make it work exactly as you wanted by now, is just because is not yet implemented.
#2. No, there's no option that indicates "track only these items but not these others", if you git-enabled your workspace, all your items (or the supported ones) will be tracked by code. You can definetly build a custom version that, via the API, deploys exactly what you want, but it takes A LOT OF TIME AND EFFORT (belive me, I've done something close to that) and you need to evaluate if the benefit outwages the effort.
#3. It think is somehow the same rational I gave in the past two questions, your best course of action is to reach out the product team and flag this, perhaps as a bug in the platform via Support / Product support.
All the very best and good luck!
svenchio I understand your point, but I think community forums are exactly the right place to discuss these kinds of platform limitations or unexpected behaviors. Not everyone has access to Microsoft Product Support.
Sharing issues like this helps identify patterns and possible workarounds across users.
In this case, the issue seems to be a real Git Sync limitation, and confirming that helps us plan around it until Microsoft provides an official fix.- svenchio
Hi crisvaillant I agree 100% > "community forums are exactly the right place to discuss these kinds of platform limitations or unexpected behaviors" 👍
And I AM NOT of the privileged people with "access to Microsoft Product Support Sharing issues like this" hence my suggestion of reporting this as an idea "ability to select with items you want to sync" and a platform issue "[...] Git Sync limitation with Airflow"
You already decribed the workaround > "temporarily ask developers to remove the Airflow job item from their incoming changes before merging/pulling" until Microsoft provides an official fix.
Hopefully, other more creative people or with some insider information on Fabric can provide additional alternatives 🤞 best of lucks mate!
BTW. Missed to kudos for your complete an amazingly well description of the issue!!! 🙏 that helps a lot!
