Dataflows - Full Loading Questions

[Kwarre35]

Is it possible that a backend environment like SAP HANA would see any load improvements For A Full Load Scenario by switching the incremental load setting to 8 quarters instead of 2 years? Has anyone tested to see if there is any difference if you alter the grain selection (from Year To Quarter To Months) of Time, but keep close to the same time frame equivalent?

[v-yueyunzh-msft · Accepted solution]

Hi , @Kwarre35 

Based on my research, there is a difference in the size of the partitions generated for the oldest data when you alter the grain selection of Time in the incremental data refresh settings for the Power BI dataset. This is because Some data sources (Azure Data Explorer, Log Analytics, Application Insight) have a limit of 64MB (compressed) on the data that can be returned from an external tool. In some case (Azure Data Explorer) an explicit error is returned, but on other cases (Log Analytics, Application Insight) the data re just truncated with no error.

For example, if you select Quarter, then the oldest partitions will be a maximum of 3 months in size, but this requires that 3 months of data still fit in the 64MB range of the data source. If you select Months, then the oldest partitions will be a maximum of 1 month in size, which would permit staying in the 64MB limit range of the data source that could extract.

In my opinion, if your data amount is too large for both of the historical data and the current data, my recommendation is to alter the grain selection to Months instead of year/quarters in order to make sure every partition has the healthy amount of data size

For more info, please refer to this official link:

Incremental refresh for datasets and real-time data in Power BI - Power BI | Microsoft Learn

Thank you for your time and sharing, and thank you for your support and understanding of PowerBI! 

Best Regards,

Aniya Zhang

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly