Forum Discussion
Notebook execution failure due to bad node
Hello Anonymous Thanks a lot for your response. The issue didnt resolve, please check below details -
Use case :
- Date size: 10TB Retail Data distributed across 24 delta tables in Lakehouse, 4 tables more than 1TB size
- Spark SQL Notebook - Select queries
- Configurations tested:
- Capacity: F128 (Number of nodes 96 – Max utilized)
- Spark cluster: Runtime 1.2(Spark 3.4, Delta 2.4)
- Node size – Medium, Large, X Large, XX Large (Same issue with All)
- Autoscale enabled
- Capacity F256 (Number of nodes 192 – Max utilized)
- Spark cluster: Runtime 1.2(Spark 3.4, Delta 2.4)
- Node size – Medium, Large, X Large, XX Large (Same issue with All)
- Autoscale enabled
- Execution: Spark SQL Notebook (Analytical queries) against lakehouse
- Issues:
- Bad Node : Spark_System_Executor_ExitCode137BadNode - Job aborted due to stage failure: Task 167 in stage 2082.0 failed 4 times, most recent failure: Lost task 167.3 in stage 2082.0 (TID 594169) (vm-e5657433 executor 274): ExecutorLostFailure (executor 274 exited caused by one of the running tasks) Reason: Container from a bad node: container_1720586556487_0001_01_000296 on host: vm-e5657433.
- Executer lost: The job failed because the executor that it was running on was lost. This may happen because the task crashed the JVM. ExitCode 137 is when a container (Spark executor) runs out of memory, YARN automatically kills it. Possible causes: 1. Driver memory issues 2. Executor Memory issues 3. Executor lost Please check the logs and see if increasing Spark partitions, number of spark partitions or reducing number of executor cores.
We are also seeing lots of 137 errors. It is frustrating on fabric that we have no visibility to the yarn diagnostics. I think it is a major blind spot and creates a lot of confusion.
We shouldn't be getting meaningless errors based on yarn configurations and yarn logs that aren't available to us. Spark is complex enough as it is, but when Fabric puts blinders on us, it becomes pretty frustrating to use. Some of us are trying to run mission-critical workloads in here, and can't even see the yarn logs.
Anyway, for those of you who aren't convinced that 137 means a yarn memory error ("early OOM"), then you should google for it. This behavior has been part of YARN/spark for a very long time, in a lot of platforms.
You should also start monitoring the "executors" tab in the spark UI, and check the boxes at the top to see the heap usage of executors. If you are exhausting 30 or 40 or 50 GB of ram in your executors, then you will surely be able to see that in realtime as it happens! For us the problem was this stupid feature called "optimized delta writes". It consumes a MASSIVE amount of ram (20 x the amount of data in parquet), and needs to be avoided whenever executors are crashing.
I finished the MT CSS support case (pro). The engineer is Chirag on Deepak's team in the Eastern US timezone.
They have a way to use kusto logs to retrieve yarn messages. Unfortunately they wouldn't share the kusto query syntax. And they say the telemetry logs are internal, in any case.
Below is the message that they say they retrieved. Obviously they are able to retrieve log data directly from yarn, unlike their customers. The following is verbatim from Chirag.
- When the memory limit is reached, the container is terminated.
2025-10-21 23:05:16,763 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: root, capacity=1.0, absoluteCapacity=1.0, maxCapacity=1.0, absoluteMaxCapacity=1.0, state=RUNNING, acls=SUBMIT_APP:*ADMINISTER_QUEUE:*, labels=*,
This indicates the capacity reached 100%.
Hopefully this is helpful. I'm still not satisfied that customers are blindfolded when we encounter yarn-related failures.