Need your help on the latest features of Power BI on Aggregations & Dual Storage Mode as we are facing some challenges. Providing the context and more details below here..
We are building a Power BI Dashboard sourcing from Databricks Delta tables and using the latest features of Power BI Aggregations by creating the Aggregation table with Dual Storage Mode options etc., We have a source data set table of size which is more than 5 GB and we have used the Direct Query Storage mode to pull the data from this table. While creating the Aggregation table by eliminating few columns and aggregating the data at few required columns granularity, while applying the query changes we have come across the below error mentioning the Total size of serialized results of tasks (4.1 GB) is bigger than spark.drive.maxResultSize (4.0 GB).
We understand that this is size limit of the Spark Driver to Import the data since the Aggregation table uses the Import Storage Mode. Need your help & suggestions to resolve this issue ? If there are any alternatives / work arounds to support this type of volume data please let us know ?
Would this issue could be resolved if we opt for the Power BI Premium Nodes (P1) with additional Capacity ?
