Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

Earn a 50% discount on the DP-600 certification exam by completing the Fabric 30 Days to Learn It challenge.

Reply
Kwarre35
Helper I
Helper I

Dataflows - Full Loading Questions

Is it possible that a backend environment like SAP HANA would see any load improvements For A Full Load Scenario by switching the incremental load setting to 8 quarters instead of 2 years? Has anyone tested to see if there is any difference if you alter the grain selection (from Year To Quarter To Months) of Time, but keep close to the same time frame equivalent?

Kwarre35_0-1684961882919.png

 

1 ACCEPTED SOLUTION
v-yueyunzh-msft
Community Support
Community Support

Hi , @Kwarre35 

Based on my research, there is a difference in the size of the partitions generated for the oldest data when you alter the grain selection of Time in the incremental data refresh settings for the Power BI dataset. This is because Some data sources (Azure Data Explorer, Log Analytics, Application Insight) have a limit of 64MB (compressed) on the data that can be returned from an external tool. In some case (Azure Data Explorer) an explicit error is returned, but on other cases (Log Analytics, Application Insight) the data re just truncated with no error.

 

 

For example, if you select Quarter, then the oldest partitions will be a maximum of 3 months in size, but this requires that 3 months of data still fit in the 64MB range of the data source. If you select Months, then the oldest partitions will be a maximum of 1 month in size, which would permit staying in the 64MB limit range of the data source that could extract.

 

In my opinion, if your data amount is too large for both of the historical data and the current data, my recommendation is to alter the grain selection to Months instead of year/quarters in order to make sure every partition has the healthy amount of data size

 

For more info, please refer to this official link:

Incremental refresh for datasets and real-time data in Power BI - Power BI | Microsoft Learn

 

Thank you for your time and sharing, and thank you for your support and understanding of PowerBI! 

 

Best Regards,

Aniya Zhang

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly

View solution in original post

1 REPLY 1
v-yueyunzh-msft
Community Support
Community Support

Hi , @Kwarre35 

Based on my research, there is a difference in the size of the partitions generated for the oldest data when you alter the grain selection of Time in the incremental data refresh settings for the Power BI dataset. This is because Some data sources (Azure Data Explorer, Log Analytics, Application Insight) have a limit of 64MB (compressed) on the data that can be returned from an external tool. In some case (Azure Data Explorer) an explicit error is returned, but on other cases (Log Analytics, Application Insight) the data re just truncated with no error.

 

 

For example, if you select Quarter, then the oldest partitions will be a maximum of 3 months in size, but this requires that 3 months of data still fit in the 64MB range of the data source. If you select Months, then the oldest partitions will be a maximum of 1 month in size, which would permit staying in the 64MB limit range of the data source that could extract.

 

In my opinion, if your data amount is too large for both of the historical data and the current data, my recommendation is to alter the grain selection to Months instead of year/quarters in order to make sure every partition has the healthy amount of data size

 

For more info, please refer to this official link:

Incremental refresh for datasets and real-time data in Power BI - Power BI | Microsoft Learn

 

Thank you for your time and sharing, and thank you for your support and understanding of PowerBI! 

 

Best Regards,

Aniya Zhang

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly

Helpful resources

Announcements
RTI Forums Carousel3

New forum boards available in Real-Time Intelligence.

Ask questions in Eventhouse and KQL, Eventstream, and Reflex.

LearnSurvey

Fabric certifications survey

Certification feedback opportunity for the community.

Top Solution Authors
Top Kudoed Authors