Notebook execution failure due to bad node

Hello Anonymous  Thanks a lot for your response. The issue didnt resolve, please check below details - 

Use case :

• Date size: 10TB Retail Data distributed across 24 delta tables in Lakehouse, 4 tables more than 1TB size
• Spark SQL Notebook - Select queries
• Configurations tested:
1. Capacity: F128 (Number of nodes 96 – Max utilized)
• Spark cluster: Runtime 1.2(Spark 3.4, Delta 2.4)
• Node size – Medium, Large, X Large, XX Large (Same issue with All)
• Autoscale enabled
2. Capacity F256 (Number of nodes 192 – Max utilized)
• Spark cluster: Runtime 1.2(Spark 3.4, Delta 2.4)
• Node size – Medium, Large, X Large, XX Large (Same issue with All)
• Autoscale enabled
3. Execution: Spark SQL Notebook (Analytical queries) against lakehouse
• Issues:
1. Bad Node : Spark_System_Executor_ExitCode137BadNode - Job aborted due to stage failure: Task 167 in stage 2082.0 failed 4 times, most recent failure: Lost task 167.3 in stage 2082.0 (TID 594169) (vm-e5657433 executor 274): ExecutorLostFailure (executor 274 exited caused by one of the running tasks) Reason: Container from a bad node: container_1720586556487_0001_01_000296 on host: vm-e5657433.
2. Executer lost: The job  failed because the executor that it was running on was lost. This may happen because the task crashed the JVM. ExitCode 137 is when a container (Spark executor) runs out of memory, YARN automatically kills it. Possible causes: 1. Driver memory issues 2. Executor Memory issues 3. Executor lost Please check the logs and see if increasing Spark partitions, number of spark partitions or reducing number of executor cores.