Forum Discussion
Intermittent CeleryExecutor state mismatches and missing task logs in Fabric Airflow
We are experiencing intermittent scheduled-task failures in a managed Microsoft Fabric Airflow environment. The failures appear to occur before normal task execution, and the failed attempts do not produce usable task or cluster logs.
Environment:
Microsoft Fabric managed Airflow
Region: North Europe
Executor: CeleryExecutor
Executor parallelism: 15
Operator: KubernetesPodOperator
Scheduled DAG runs
Observed behaviour:
For one scheduled run, the Airflow Audit Log contained a message similar to the following:
Executor CeleryExecutor(parallelism=15) reported that the task instance for an affected DAG task, in queued state, finished with state failed, but the task instance state attribute was still queued.
The same state mismatch occurred for several unrelated tasks within the same DAG run.
The affected tasks use KubernetesPodOperator. Some run on Fabric-managed Kubernetes infrastructure, while another targets an external AKS cluster. Because different workloads are affected simultaneously, this may indicate an issue involving the Celery worker, executor, scheduler, broker, or task-state synchronization rather than a problem with a particular container image or source system.
Missing task logs:
After the tasks retried, opening the failed attempt logs produced this error:
Could not read served logs. Invalid URL http://:8793/log/dag_id=.../attempt=2.log. No host supplied.
The generated log URL has no hostname.
The Fabric Cluster Logs view also had no entries for the incident period. As a result, we cannot determine:
Whether a Celery worker received or started the task
Whether the Kubernetes pod was created
Why the executor reported the queued task as failed
Why the worker hostname was missing from the task log URL
Retries sometimes succeed, so the issue is intermittent.
Questions:
What can cause CeleryExecutor to report a task as failed while the task instance remains in the queued state?
Does a blank hostname in the served-log URL indicate that the Celery worker exited before its hostname was recorded?
Are there additional Fabric logs or monitoring locations that show Celery worker lifecycle, heartbeat, broker connectivity, scheduler activity, or Kubernetes pod creation?
Are there known limitations or reliability issues when using KubernetesPodOperator with Fabric managed Airflow?
Any guidance on additional diagnostics or configuration checks would be useful.
3 Replies
Hi sahinad,
Based on the Microsoft Fabric documentation, this behavior isn't a documented limitation of Fabric managed Airflow, and there is no supported way to access internal Celery worker, broker, or scheduler logs. The available diagnostics are the Monitoring Hub, Workspace Monitoring (if enabled), Airflow task logs, and cluster logs, and there are no documented limitations specific to KubernetesPodOperator
or the blank served-log hostname. If the issue persists with missing logs, please open a Microsoft Fabric support ticket and share the workspace ID, region, DAG run IDs, timestamps, and correlation/activity IDs for further backend investigation.Create a Fabric and Power BI Support Ticket - Power BI | Microsoft Learn
Thank you.
Thanks for response. I'll create a support ticket right away.
Based on your description, this sounds more like an infrastructure or executor synchronization issue than a problem with your DAG or KubernetesPodOperator implementation.
A few thoughts on your questions:
1. Queued → Failed state mismatch
This can occur if the Celery worker, scheduler, or broker loses synchronization. For example, if a worker crashes, loses broker connectivity, or times out before acknowledging the task, the executor may report the task as failed while the scheduler still considers it queued until reconciliation occurs.
2. Blank hostname in the log URL
A missing hostname typically suggests the task never reached a healthy worker, or the worker exited before its metadata was registered. Since the log URL has no host, it appears the log-serving endpoint was never successfully associated with that task attempt.
3. Additional diagnostics
Unfortunately, Fabric's managed Airflow environment doesn't expose all of the underlying Celery, broker, or Kubernetes infrastructure logs that you would have access to in a self-managed deployment. If both the Airflow task logs and Fabric Cluster Logs are empty during the incident, there may be little additional telemetry available from the user side. It would be useful to correlate:
Airflow scheduler logs around the failure time.
Workspace capacity health or service incidents.
Whether multiple unrelated DAGs experienced similar behavior at the same time.
4. KubernetesPodOperator
KubernetesPodOperator is supported, but the behavior you're seeing doesn't point to the operator itself, especially since it affects both Fabric-managed Kubernetes and an external AKS cluster. That suggests the issue is occurring before pod creation or task dispatch rather than within the Kubernetes workloads.
Since the failures are intermittent, affect multiple unrelated tasks simultaneously, produce invalid log URLs, and leave no cluster logs, I'd recommend opening a Microsoft support ticket. Include the DAG run IDs, timestamps, workspace/region, and evidence that retries succeed while initial attempts fail. That should help Microsoft determine whether there was a transient issue with the managed Celery workers, scheduler, broker, or Fabric infrastructure during those execution windows.
