Forum Discussion
Memorable MemberSpark Cluster's scheduler is killing container for some reason
I finished the MT CSS support case (pro). The engineer is Chirag on Deepak's team in the Eastern US timezone.
They have a way to use kusto logs to retrieve yarn messages. Unfortunately they wouldn't share the kusto query syntax. And they say the telemetry logs are internal, in any case.
Below is the message that they say they retrieved. Obviously they are able to retrieve log data directly from yarn, unlike their customers. The following is verbatim from Chirag.
- When the memory limit is reached, the container is terminated.
2025-10-21 23:05:16,763 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: root, capacity=1.0, absoluteCapacity=1.0, maxCapacity=1.0, absoluteMaxCapacity=1.0, state=RUNNING, acls=SUBMIT_APP:*ADMINISTER_QUEUE:*, labels=*,
This indicates the capacity reached 100%.
Hopefully this is helpful. I'm still not satisfied that customers are blindfolded when we encounter yarn-related failures.
Community SupportHi dbeavon3,
We would like to follow up and see whether the details we shared have resolved your problem.
If you need any more assistance, please feel free to connect with the Microsoft Fabric community.
Thank you.
Memorable MemberNo it doesn't help. The reason I posted was to get a specific diagnosis for the killing of an executor and I'm not any closer to that. Since it appears to be deliberately killed by this spark environment, then the REASON for that should be discoverable. But I haven't found the messages in the logs saying WHY spark killed my executor, and you haven't shown me where to look for those messages.