Forum Discussion

mjohannesson's avatar
mjohannesson
Icon for Advocate I rankAdvocate I
6 years ago

Refresh fails for large datasets using Spark connector

Our reports and datasets imports data from Databricks Spark Delta tables using the Spark connector into our Premium P1 capacity. We're using incremental refresh for the larger (fact) tables, but we'r...
Need Help
promagnus's avatar
promagnus
New Member
6 years ago

Hi Anonymous ,

 

Thanks for the input. In the Advanced Options section of the Azure Databricks cluster I use for data refresh, the Spark Config is already set to "spark.driver.maxResultSize 0", which should mean it is unlimited, but still some of the error messages claims that is's set to 4.0 GB. Why is that?

 

You also mentioned the timeout properties for database connection and Power BI service. Where can I find these?

 

Thanks,
Magnus

Anonymous's avatar
Anonymous
Not applicable
6 years ago

HI promagnus,

#1, It sounds like power bi connector still uses the default setting instead of data source setting, perhaps you can contact to power bi team to confirm this.
#2, It means the max available timeout of the datasource connection, you can check the cluster configuration to find out the related properties of session timeout.

Cluster configurations#spark-config 

Regards,

Xiaoxin Sheng