Forum Discussion

MiSchroe's avatar
MiSchroe
Frequent Visitor
2 years ago

Performance issue with Spark structured streaming job

Hi,   we face some performance issues when running a Spark Job Definition with a structured streaming job getting data from a Kafka. We just want to dump the Kafka data into a Lakehouse for further...
data engineering
MiSchroe's avatar
MiSchroe
Frequent Visitor
2 years ago

Hi Anonymous,

 

thanks for your answer.

 

I have implemented your suggestions and here are the results:

  1. Spark executor heartbeat -> no change.
  2. Increased driver and 3 executors to X-large nodes with each 32 cores and 224GB memory. -> no change.
  3. Increase the number of Kafka partitions: As it is an external system I can't change that one. There are 6 partitions on that Kafka topic. (And it runs perfectly with this setup in Databricks, so the number of partitions is not an issue here.)
  4. Resize the executor: see #2 -> no change.

The topic produces messages at a rate of ~10 messages per second with each meassage ~3kB in size. Compared to what is mentioned in the linked blog entry this is a drop in the ocean.

 

Regards,

Michael