Forum Discussion

MiSchroe's avatar
MiSchroe
Frequent Visitor
2 years ago

Performance issue with Spark structured streaming job

Hi,   we face some performance issues when running a Spark Job Definition with a structured streaming job getting data from a Kafka. We just want to dump the Kafka data into a Lakehouse for further...
data engineering
Anonymous's avatar
Anonymous
Not applicable
2 years ago

Hi MiSchroe ,

 

Given that the same code runs efficiently on Databricks, the problem may be in the configuration or environment of the current setup.

 

I have the following suggestions for performance tuning:

 

It is recommended to increase spark.executor.heartbeatInterval from 10 seconds to 20 seconds.

 

Increase driver and executor memory.

 

Increase parallelism: increase the number of partitions in Kafka.

 

Correctly resize the executor.

 

More details on performance tuning can be found in this article:

Performance Tuning of an Apache Kafka/Spark Streaming System | HPE Developer Portal

 

Best Regards,
Yang
Community Support Team

 

If there is any post helps, then please consider Accept it as the solution  to help the other members find it more quickly.
If I misunderstand your needs or you still have problems on it, please feel free to let us know. Thanks a lot!