Py4JJavaError:

Hi tan_thiamhuat ,

This error usually comes up in Spark when the data or objects you’re sending between nodes are too large for the default configuration. The key setting here is spark.rpc.message.maxSize, and you can bump it up directly in your notebook.

In your Python notebook, you can increase this limit by adding a cell at the top with the following:

%%configure -f
{
    "conf": {
        "spark.rpc.message.maxSize": "512"
    }
}

You can adjust the value (like 512) to something higher if needed, depending on your data size.

If you’re still hitting limits after increasing this, it’s often a good idea to refactor your code to avoid sending huge objects between nodes, maybe by using broadcast variables or splitting up the data.

Hope this helps! Let us know if you run into any more issues.