Notebook execution failure due to bad node

Question

Hello All,We are currently using F128 capacity and Spark pool configuration: Runtime 1.2 (Spark 3.4 Delta 2.4), Compute Medium, 1-95 node.&nbsp;The issue is for compute exhaustive queries, the notebook execution is getting failed with the following error, any help will be highly appreciated.&nbsp;Job aborted due to stage failure: Task 167 in stage 2082.0 failed 4 times, most recent failure: Lost task 167.3 in stage 2082.0 (TID 594169) (vm-e5657433 executor 274): ExecutorLostFailure (executor 274 exited caused by one of the running tasks) Reason: Container from a bad node: container_1720586556487_0001_01_000296 on host: vm-e5657433. Exit status: 137. Diagnostics: [2024-07-10 05:41:58.081]Container killed on request. Exit code is 137[2024-07-10 05:41:58.098]Container exited with a non-zero exit code 137.[2024-07-10 05:41:58.101]Killed by external signal

abhradwip · Answer

Hello&nbsp;Anonymous&nbsp; Thanks a lot for your response. The issue didnt resolve, please check below details -&nbsp;&nbsp;Use case :Date size: 10TB Retail Data distributed across 24 delta tables in Lakehouse, 4 tables more than 1TB sizeSpark SQL Notebook - Select queriesConfigurations tested:Capacity: F128 (Number of nodes 96 – Max utilized)Spark cluster: Runtime 1.2(Spark 3.4, Delta 2.4)Node size – Medium, Large, X Large, XX Large (Same issue with All)Autoscale enabledCapacity F256 (Number of nodes 192 – Max utilized)Spark cluster: Runtime 1.2(Spark 3.4, Delta 2.4)Node size – Medium, Large, X Large, XX Large&nbsp;(Same issue with All)Autoscale enabledExecution: Spark SQL Notebook (Analytical queries) against lakehouseIssues:Bad Node : Spark_System_Executor_ExitCode137BadNode - Job aborted due to stage failure: Task 167 in stage 2082.0 failed 4 times, most recent failure: Lost task 167.3 in stage 2082.0 (TID 594169) (vm-e5657433 executor 274): ExecutorLostFailure (executor 274 exited caused by one of the running tasks) Reason: Container from a bad node: container_1720586556487_0001_01_000296 on host: vm-e5657433.Executer lost: The job &nbsp;failed because the executor that it was running on was lost. This may happen because the task crashed the JVM. ExitCode 137 is when a container (Spark executor) runs out of memory, YARN automatically kills it. Possible causes: 1. Driver memory issues 2. Executor Memory issues 3. Executor lost Please check the logs and see if increasing Spark partitions, number of spark partitions or reducing number of executor cores.

felixl · Answer

Hi! Were you able to find a solution to your issue? am facing similar issues when migrating jobs from Azure Synapse to Fabric The jobs run fine in Synapse, but I get "bad node" executor lost error messages left and right in Fabric. No idea why... (memory and CPU is not satuated at the time of executor crashing)

Thanks!

anonymous · Answer

Hi&nbsp;abhradwip&nbsp;,
The error message "Exit code is 137" usually means that the container exceeded its memory limit or was manually terminated. There are several steps you can take to find the problem:

Monitor memory usage in the Spark UI to determine if certain nodes are out of memory.
Consider increasing the memory allocation for the executable.
Check the operational status and state of the nodes identified in the error message "vm-e5657433".

If it does not help,&nbsp;you can provide me with your code so that I can find out if you are having problems.&nbsp;In the meantime you can provide some details, for example, did it run normally before and suddenly get an error today. If so, did you do anything else before running it.
&nbsp;
Best Regards,
Ada Wang
If this post&nbsp;helps, then please consider&nbsp;Accept it as the solution&nbsp;to help the other members find it more quickly.
&nbsp;

anonymous · Answer

Hi&nbsp;abhradwip&nbsp;,
Can I ask if your problem is resolved? It looks as if it is an intermittent issue, you might consider updating your drivers, clearing your cache and then refreshing your browser and re-running your Notebook to see if the issue is resolved.
&nbsp;
Best Regards,
Ada Wang
If this post&nbsp;helps, then please consider&nbsp;Accept it as the solution&nbsp;to help the other members find it more quickly.

dbeavon3 · Answer

We are also seeing lots of 137 errors.&nbsp; It is frustrating on fabric that we have no visibility to the yarn diagnostics.&nbsp; I think it is a major blind spot and creates a lot of confusion.&nbsp;We shouldn't be getting meaningless errors based on yarn configurations and yarn logs that aren't available to us.&nbsp; Spark is complex enough as it is, but when Fabric puts blinders on us, it becomes pretty frustrating to use.&nbsp; Some of us are trying to run mission-critical workloads in here, and can't even see the yarn logs.&nbsp;&nbsp;Anyway, for those of you who aren't convinced that 137 means a yarn memory error ("early OOM"), then you should google for it.&nbsp; This behavior has been part of YARN/spark for a very long time, in a lot of platforms.&nbsp;You should also start monitoring the "executors" tab in the spark UI, and check the boxes at the top to see the heap usage of executors.&nbsp; If you are exhausting 30 or 40 or 50 GB of ram in your executors, then you will surely be able to see that in realtime as it happens!&nbsp; For us the problem was this stupid feature called "optimized delta writes".&nbsp; It consumes a MASSIVE amount of ram (20 x the amount of data in parquet), and needs to be avoided whenever executors are crashing.&nbsp;&nbsp;&nbsp;

Forum Discussion

Notebook execution failure due to bad node

6 Replies

Related content

Dependency failures blocking notebook execution

Fabric Python Notebook Execution Logs

Pipline Executing Notebook Fails, Failed to create Livy session for executing notebook

Scheduled pipeline notebook execution fails intermittently

Not receiving Failure Notification on Fabric Pipelines and Notebooks