Incremental Refresh Timeout Issue with Large Dataset

[AshokAinala]

Hi All,

I have created the RangeStart and RangeEnd parameters and applied them to the date column in the fact table as the first step for configuring incremental refresh.

When I configure incremental refresh for 3 weeks (approximately 1.8 million records), the refresh completes successfully. However, when I configure incremental refresh for 6 months (approximately 10 million records), the refresh fails with a timeout error and displays the message: "Dataset is too large, please reduce the size."

Unfortunately, reducing the dataset size is not an option, as the business requires all 10 million records.

Could someone help me understand why this timeout error is occurring with incremental refresh, even though I have configured the RangeStart and RangeEnd parameters correctly? Are there any limitations, configuration changes, or optimization techniques that I should consider for handling a dataset of this size?

Thanks, Ashok

[v-achippa]

Hi @AshokAinala,

Thank you for reaching out to Microsoft Fabric Community.

Thank you for the details. Here since the RangeStart and RangeEnd parameters are configured correctly, looks like the issue is due to the initial refresh, which needs to load all 10 million records. This is likely exceeding the available resources and causing the error

• Please verify the query folding is working for the RangeStart and RangeEnd filters, as the incremental refresh relies on query folding for efficient data loading. If query folding is working and the issue still occurs, this is likely a capacity or resource limitation. In that case, please use a Premium/Fabric capacity with an XMLA endpoint bootstrap refresh for the initial load.

For your reference, please follow below documentation:

Configure incremental refresh and real-time data for Power BI semantic models - Power BI | Microsoft...

Advanced Incremental Refresh and Real-Time Data With the XMLA Endpoint in Power BI - Power BI | Micr...

Thanks and regards,

Anjan Kumar Chippa