Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

Find everything you need to get certified on Fabric—skills challenges, live sessions, exam prep, role guidance, and more. Get started

Reply
Kwarre35
Helper I
Helper I

Dataflows - Full Loading Questions

Is it possible that a backend environment like SAP HANA would see any load improvements For A Full Load Scenario by switching the incremental load setting to 8 quarters instead of 2 years? Has anyone tested to see if there is any difference if you alter the grain selection (from Year To Quarter To Months) of Time, but keep close to the same time frame equivalent?

Kwarre35_0-1684961882919.png

 

1 ACCEPTED SOLUTION
v-yueyunzh-msft
Community Support
Community Support

Hi , @Kwarre35 

Based on my research, there is a difference in the size of the partitions generated for the oldest data when you alter the grain selection of Time in the incremental data refresh settings for the Power BI dataset. This is because Some data sources (Azure Data Explorer, Log Analytics, Application Insight) have a limit of 64MB (compressed) on the data that can be returned from an external tool. In some case (Azure Data Explorer) an explicit error is returned, but on other cases (Log Analytics, Application Insight) the data re just truncated with no error.

 

 

For example, if you select Quarter, then the oldest partitions will be a maximum of 3 months in size, but this requires that 3 months of data still fit in the 64MB range of the data source. If you select Months, then the oldest partitions will be a maximum of 1 month in size, which would permit staying in the 64MB limit range of the data source that could extract.

 

In my opinion, if your data amount is too large for both of the historical data and the current data, my recommendation is to alter the grain selection to Months instead of year/quarters in order to make sure every partition has the healthy amount of data size

 

For more info, please refer to this official link:

Incremental refresh for datasets and real-time data in Power BI - Power BI | Microsoft Learn

 

Thank you for your time and sharing, and thank you for your support and understanding of PowerBI! 

 

Best Regards,

Aniya Zhang

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly

View solution in original post

1 REPLY 1
v-yueyunzh-msft
Community Support
Community Support

Hi , @Kwarre35 

Based on my research, there is a difference in the size of the partitions generated for the oldest data when you alter the grain selection of Time in the incremental data refresh settings for the Power BI dataset. This is because Some data sources (Azure Data Explorer, Log Analytics, Application Insight) have a limit of 64MB (compressed) on the data that can be returned from an external tool. In some case (Azure Data Explorer) an explicit error is returned, but on other cases (Log Analytics, Application Insight) the data re just truncated with no error.

 

 

For example, if you select Quarter, then the oldest partitions will be a maximum of 3 months in size, but this requires that 3 months of data still fit in the 64MB range of the data source. If you select Months, then the oldest partitions will be a maximum of 1 month in size, which would permit staying in the 64MB limit range of the data source that could extract.

 

In my opinion, if your data amount is too large for both of the historical data and the current data, my recommendation is to alter the grain selection to Months instead of year/quarters in order to make sure every partition has the healthy amount of data size

 

For more info, please refer to this official link:

Incremental refresh for datasets and real-time data in Power BI - Power BI | Microsoft Learn

 

Thank you for your time and sharing, and thank you for your support and understanding of PowerBI! 

 

Best Regards,

Aniya Zhang

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly

Helpful resources

Announcements
Europe Fabric Conference

Europe’s largest Microsoft Fabric Community Conference

Join the community in Stockholm for expert Microsoft Fabric learning including a very exciting keynote from Arun Ulag, Corporate Vice President, Azure Data.

Power BI Carousel June 2024

Power BI Monthly Update - June 2024

Check out the June 2024 Power BI update to learn about new features.

RTI Forums Carousel3

New forum boards available in Real-Time Intelligence.

Ask questions in Eventhouse and KQL, Eventstream, and Reflex.

Top Solution Authors