Forum Discussion
Py4JJavaError: An error occurred while calling o84455.csv.
Thanks for clarifying! So you’re using coalesce(1) because you need a single CSV file, then you move it to SFTP. That makes sense.
But keep in mind:
- Using coalesce(1) on a big DataFrame forces all the data to one node/worker, which can cause memory issues or serialization errors—especially with large datasets like yours.
- That’s usually why the Spark job fails or throws those message size/serialization errors.
Possible solutions:
If you must have a single file, try to:
- Increase spark.rpc.message.maxSize even more (if you haven’t already).
- Make sure your cluster has enough memory/resources for one node to handle the whole DataFrame.
- If possible, filter or reduce your data before doing coalesce(1) to make the final file smaller.
Alternative approach (if you keep hitting errors):
- Write the CSV without coalesce(1) (so you get multiple part files).
- Combine those part files into one CSV outside of Spark (with a shell script, Python, etc.) before SFTP transfer.
Let me know if you still get errors or want help with merging the part files after export!
Hi tan_thiamhuat ,
You are getting this error because the size of the serialized task is much larger than the value allowed by spark.rpc.message.maxSize. Even though you set spark.rpc.message.maxSize to 2048, it might still not be enough for your data, or the configuration may not be applied correctly everywhere in your cluster.
Here’s what you can try:
- Make sure you are setting spark.rpc.message.maxSize in both your SparkSession and in your spark-submit command, like this:
spark = SparkSession.builder \
.appName("Increase RPC Message Size") \
.config("spark.rpc.message.maxSize", "4096") \
.getOrCreate()And when submitting your job:
spark-submit --conf spark.rpc.message.maxSize=4096 ...
Try increasing the value even more if necessary, but keep in mind that very large values can cause instability or memory issues on your cluster. If possible, break your data into smaller partitions or avoid collecting very large objects at once.
Make sure all nodes (driver and workers) have the same configuration value for spark.rpc.message.maxSize.
If you are sending very large DataFrames or objects, try to process them in smaller chunks or use broadcast variables only for reasonably sized data.
After updating the configs, restart your Spark session or cluster to ensure settings are applied.
If you still get this error, please share more details about when the error happens (for example, during collect(), join, or broadcast), so I can give more targeted advice.