Py4JJavaError:

[tan_thiamhuat]

Py4JJavaError: An error occurred while calling o32067.csv. : org.apache.spark.SparkException: Job aborted due to stage failure: Serialized task 182:0 was 197125878 bytes, which exceeds max allowed: spark.rpc.message.maxSize (134217728 bytes). Consider increasing spark.rpc.message.maxSize or using broadcast variables for large values.

Where or what settings can I do in the Python notebook to increase its size?

Thanks.

[Anonymous]

Hi @tan_thiamhuat,

As we haven’t heard back from you, we would like to follow up to see if the solution provided by the super user resolved your issue. Please let us know if you need any further assistance.
If our super user response resolved your issue, please mark it as "Accept as solution" and click "Yes" if you found it helpful.

Regards,
Vinay Pabbu

[burakkaragoz · Accepted solution]

Hi @tan_thiamhuat ,

This error usually comes up in Spark when the data or objects you’re sending between nodes are too large for the default configuration. The key setting here is spark.rpc.message.maxSize, and you can bump it up directly in your notebook.

In your Python notebook, you can increase this limit by adding a cell at the top with the following:

%%configure -f
{
    "conf": {
        "spark.rpc.message.maxSize": "512"
    }
}

You can adjust the value (like 512) to something higher if needed, depending on your data size.

If you’re still hitting limits after increasing this, it’s often a good idea to refactor your code to avoid sending huge objects between nodes, maybe by using broadcast variables or splitting up the data.

Hope this helps! Let us know if you run into any more issues.

[nilendraFabric]

Hello @tan_thiamhuat 

This means a Spark job tried to send a message (such as a serialized task or data) that is larger than the configured maximum (`spark.rpc.message.maxSize`, default 128 MiB).

%%configure -f
{
"conf": {
"spark.rpc.message.maxSize": "512"
}
}

try this too

df = df.repartition(100) # Increase the number as needed