DataBricks error

[NimaiAhluwalia]

OLE DB or ODBC error: [DataSource.Error] ODBC: ERROR [HY000] [Microsoft][Hardy] (35) Error from server: error code: '0' error message: 'Error running query: org.apache.spark.SparkException: Job aborted due to stage failure: Total size of serialized results of 142 tasks (20.1 GB) is bigger than spark.driver.maxResultSize (20.0 GB)

Regards

[v-angzheng-msft · Accepted solution]

Hi,   @NimaiAhluwalia 

You need to change this parameter in the cluster configuration. Go into the cluster settings, under Advanced select spark and paste spark.driver.maxResultSize 0 (for unlimited) or whatever the value suits you. Using 0 is not recommended. You should optimize the job by re partitioning.

See the links below for more information:

https://docs.microsoft.com/en-us/azure/databricks/kb/jobs/job-fails-maxresultsize-exception#solution

https://stackoverflow.com/questions/53067556/databricks-exception-total-size-of-serialized-results-i...

https://stackoverflow.com/questions/31058504/spark-1-4-increase-maxresultsize-memory

https://stackoverflow.com/questions/47996396/total-size-of-serialized-results-of-16-tasks-1048-5-mb-...

https://stackoverflow.com/questions/46763214/total-size-of-serialized-results-of-tasks-is-bigger-tha...

https://issues.apache.org/jira/browse/SPARK-12837

Best Regards,
Community Support Team _ Zeon Zheng
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.