Forum Discussion
Notebook execution failure due to bad node
We are also seeing lots of 137 errors. It is frustrating on fabric that we have no visibility to the yarn diagnostics. I think it is a major blind spot and creates a lot of confusion.
We shouldn't be getting meaningless errors based on yarn configurations and yarn logs that aren't available to us. Spark is complex enough as it is, but when Fabric puts blinders on us, it becomes pretty frustrating to use. Some of us are trying to run mission-critical workloads in here, and can't even see the yarn logs.
Anyway, for those of you who aren't convinced that 137 means a yarn memory error ("early OOM"), then you should google for it. This behavior has been part of YARN/spark for a very long time, in a lot of platforms.
You should also start monitoring the "executors" tab in the spark UI, and check the boxes at the top to see the heap usage of executors. If you are exhausting 30 or 40 or 50 GB of ram in your executors, then you will surely be able to see that in realtime as it happens! For us the problem was this stupid feature called "optimized delta writes". It consumes a MASSIVE amount of ram (20 x the amount of data in parquet), and needs to be avoided whenever executors are crashing.
I finished the MT CSS support case (pro). The engineer is Chirag on Deepak's team in the Eastern US timezone.
They have a way to use kusto logs to retrieve yarn messages. Unfortunately they wouldn't share the kusto query syntax. And they say the telemetry logs are internal, in any case.
Below is the message that they say they retrieved. Obviously they are able to retrieve log data directly from yarn, unlike their customers. The following is verbatim from Chirag.
- When the memory limit is reached, the container is terminated.
2025-10-21 23:05:16,763 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: root, capacity=1.0, absoluteCapacity=1.0, maxCapacity=1.0, absoluteMaxCapacity=1.0, state=RUNNING, acls=SUBMIT_APP:*ADMINISTER_QUEUE:*, labels=*,
This indicates the capacity reached 100%.
Hopefully this is helpful. I'm still not satisfied that customers are blindfolded when we encounter yarn-related failures.