Forum Discussion
Azure Data Bricks Data Refresh
I have a report utilizing data import from a persisted table in Databricks. Once the dataset size increased I received the following error:
Total size of serialized results of 17 tasks (4.1 GB) is bigger than spark.driver.maxResultSize
Looking up the error I found alot of spark specific posts explaining that spark.driver.maxResultSize is a variable which exists to prevent out of memory exceptions. The reason I'm posting in a Power BI forum is I haven't had any issue interacting with the data (either in munging the data or writing it to hive) on the Databricks side.
Does anybody know some details about how the refresh interacts with spark/Databricks and why it could be causing the issue in this particular situation? I would prefer having some understanding of why it's occurring in this situation before I adjust the maxResultSize variable (possibly several times).
5 Replies
Hi Anonymous ,
I think this issue should more related to database settings. it sounds like response data amount is greater than default cache size so refresh requests has been blocked/canceled.
Maybe you can take a look at following link to know more about this issue:
For Power BI Architecture, you can refer to below link:
Regards,
Xiaoxin Sheng
Hi Xiaoxin,
Thanks for looking at this issue. As an update we increased teh variable size to 35gb on both the clusters we are running, but still encounter the same 4gb error when attempting a refresh. We have a ticket in with the dev team to ascertain whether the error is being thrown by our Databricks instance (meaning we missed something in the adjustment of the variable) or whether it's occurring in the attempt to write to our premium capacity storage.
Julius
Did you manage to figure this out? I am getting the same error.
