Spark executor runs out of memory

Hi spencer_sa 

To address the Spark executor out of memory issue , you can try and test following options:

Increase the executor memory using the %%configure magic command in your notebook:

%%configure -f
{
"executorMemory": "56g"
}

Set the executor memory overhead to account for non-heap memory usage:

%%configure -f
{
"conf": {
"spark.executor.memoryOverhead": "8g"
}
}

Enable dynamic allocation to automatically adjust the number of executors:

%%configure -f
{
"conf": {
"spark.dynamicAllocation.enabled": "true"
}
}

Utilize Fabric’s Monitoring Hub and Spark History Server to identify performance bottlenecks and optimize your Spark jobs

Remember to restart your Spark session after making configuration changes. These optimizations should help mitigate executor out-of-memory errors and improve overall Spark job performance in Microsoft Fabric.

custom pool creation could be an option as well.

please see if this resolves your performance issue

if this helps please give kudos and accept the solution.

thanks