Forum Discussion

mjohannesson's avatar
mjohannesson
Icon for Advocate I rankAdvocate I
6 years ago

Refresh fails for large datasets using Spark connector

Our reports and datasets imports data from Databricks Spark Delta tables using the Spark connector into our Premium P1 capacity. We're using incremental refresh for the larger (fact) tables, but we'r...
Need Help
Anonymous's avatar
Anonymous
Not applicable
6 years ago

HI mjohannesson,

>>Job aborted due to stage failure: Total size of serialized results of 19 tasks (4.2 GB) is bigger than spark.driver.maxResultSize (4.0 GB)'.. The exception was raised by the IDbCommand interface.

Please take a look at following document about maxResultsize issue:

Apache Spark job fails with maxResultSize exception 

In addition, you can take a look at the timeout properties setting on both database connection and power bi service side to choose a suitable refresh timeout. (it may reduce the issues when power bi processing to refresh large datasets)

BTW, Please also check at the following link about the optional parameter 'batch size' if it help for your scenario.

Turbo boost data loads from Spark using SQL Spark connector 
Regards,

Xiaoxin Sheng

  • promagnus's avatar
    promagnus
    New Member
    6 years ago

    Hi Anonymous ,

     

    Thanks for the input. In the Advanced Options section of the Azure Databricks cluster I use for data refresh, the Spark Config is already set to "spark.driver.maxResultSize 0", which should mean it is unlimited, but still some of the error messages claims that is's set to 4.0 GB. Why is that?

     

    You also mentioned the timeout properties for database connection and Power BI service. Where can I find these?

     

    Thanks,
    Magnus

    • Anonymous's avatar
      Anonymous
      Not applicable
      6 years ago

      HI promagnus,

      #1, It sounds like power bi connector still uses the default setting instead of data source setting, perhaps you can contact to power bi team to confirm this.
      #2, It means the max available timeout of the datasource connection, you can check the cluster configuration to find out the related properties of session timeout.

      Cluster configurations#spark-config 

      Regards,

      Xiaoxin Sheng