Forum Discussion
tan_thiamhuat
Post Patron
Post PatronPy4JJavaError: An error occurred while calling o84455.csv.
Py4JJavaError: An error occurred while calling o84455.csv. : org.apache.spark.SparkException: Job aborted due to stage failure: Serialized task 661:0 was 145766442 bytes, which exceeds max allowed: s...
Thanks for clarifying! So you’re using coalesce(1) because you need a single CSV file, then you move it to SFTP. That makes sense.
But keep in mind:
- Using coalesce(1) on a big DataFrame forces all the data to one node/worker, which can cause memory issues or serialization errors—especially with large datasets like yours.
- That’s usually why the Spark job fails or throws those message size/serialization errors.
Possible solutions:
If you must have a single file, try to:
- Increase spark.rpc.message.maxSize even more (if you haven’t already).
- Make sure your cluster has enough memory/resources for one node to handle the whole DataFrame.
- If possible, filter or reduce your data before doing coalesce(1) to make the final file smaller.
Alternative approach (if you keep hitting errors):
- Write the CSV without coalesce(1) (so you get multiple part files).
- Combine those part files into one CSV outside of Spark (with a shell script, Python, etc.) before SFTP transfer.
Let me know if you still get errors or want help with merging the part files after export!
tan_thiamhuatPost Patron
Post Patronthis line gets the error:
df_clean.coalesce(1).write.mode("overwrite").option("header", "true").csv(onelake_path)
where df_clean is of 349327 rows and 24 columns
onelake_path = 'Files/Cleaned/3_BillingExport_20250504111740385_cleaned.csv'
burakkaragoz
Super User
Super UserIt looks like you’re hitting the error when writing out your CSV with:
Python
df_clean.coalesce(1).write.mode("overwrite").option("header", "true").csv(onelake_path)
with a pretty big DataFrame (349,927 rows, 24 columns).
If you’re still facing the same Spark error about message size or task serialization, it’s likely because:
- The data being written is too large for the default Spark RPC message size.
- The config change (spark.rpc.message.maxSize) might not be picked up by all nodes, or it may need to be increased even more.
Things to try:
- Double check you set the config before creating your SparkSession (as in my previous post).
- Try bumping up spark.rpc.message.maxSize higher (4096, 8192, etc).
- If possible, avoid coalesce(1) (writing as a single file) for huge data—it forces all data to one node, which can cause memory or serialization issues.
- Try writing without coalesce(1) and see if it works.
If you can share the exact error message you’re getting here, it may help pinpoint the issue better!