Forum Discussion

Nape's avatar
Nape
Microsoft Employee
2 years ago
Solved

Kryo serialization exception

We are using the following code in his synapse notebook:   spark = SparkSession.builder.appName("dynamicdatamerge").config("spark.serializer", "org.apache.spark.serializer.KryoSerializer" ).config(...
data engineering
data science
Anonymous's avatar
Anonymous
Not applicable
2 years ago

Hi @Nape ,

I understand that you have tried reducing the size of the object you are trying to serialize, but this has not resolved the issue.
FYI: For faster serialization and deserialization spark itself recommends to use Kryo serialization.

I have a few additional suggestions for you, can you please try to refer the below links -
Kryo Serialization in Spark - Knoldus Blogs
Apache Spark: All about Serialization | by Jay | SelectFrom
8 Performance Optimization Techniques Using Spark - Syntelli Solutions Inc.
Troubleshooting Spark Issues — Qubole Data Service documentation

Hope this helps in resolving your issue.

Thank you

Anonymous's avatar
Anonymous
Not applicable
2 years ago

Hi Nape ,

Glad to know that your query got resolved.

Please continue using Fabric Community for help regarding your issues.