Forum Discussion
Incremental Refresh Timeout Issue with Large Dataset
Hi All,
I have created the RangeStart and RangeEnd parameters and applied them to the date column in the fact table as the first step for configuring incremental refresh.
When I configure incremental refresh for 3 weeks (approximately 1.8 million records), the refresh completes successfully. However, when I configure incremental refresh for 6 months (approximately 10 million records), the refresh fails with a timeout error and displays the message: "Dataset is too large, please reduce the size."
Unfortunately, reducing the dataset size is not an option, as the business requires all 10 million records.
Could someone help me understand why this timeout error is occurring with incremental refresh, even though I have configured the RangeStart and RangeEnd parameters correctly? Are there any limitations, configuration changes, or optimization techniques that I should consider for handling a dataset of this size?
Thanks, Ashok
Hi AshokAinala,
Thank you for reaching out to Microsoft Fabric Community.
Thank you for the details. Here since the RangeStart and RangeEnd parameters are configured correctly, looks like the issue is due to the initial refresh, which needs to load all 10 million records. This is likely exceeding the available resources and causing the error
- Please verify the query folding is working for the RangeStart and RangeEnd filters, as the incremental refresh relies on query folding for efficient data loading. If query folding is working and the issue still occurs, this is likely a capacity or resource limitation. In that case, please use a Premium/Fabric capacity with an XMLA endpoint bootstrap refresh for the initial load.
For your reference, please follow below documentation:
Thanks and regards,
Anjan Kumar Chippa
3 Replies
Hi AshokAinala,
Thank you for reaching out to Microsoft Fabric Community.
Thank you for the details. Here since the RangeStart and RangeEnd parameters are configured correctly, looks like the issue is due to the initial refresh, which needs to load all 10 million records. This is likely exceeding the available resources and causing the error
- Please verify the query folding is working for the RangeStart and RangeEnd filters, as the incremental refresh relies on query folding for efficient data loading. If query folding is working and the issue still occurs, this is likely a capacity or resource limitation. In that case, please use a Premium/Fabric capacity with an XMLA endpoint bootstrap refresh for the initial load.
For your reference, please follow below documentation:
Thanks and regards,
Anjan Kumar Chippa
Hi AshokAinala,
As we haven’t heard back from you, we wanted to kindly follow up to check if the solution I have provided for the issue worked? or let us know if you need any further assistance.
Thanks and regards,
Anjan Kumar Chippa
Hi @AshokAinala,
We wanted to kindly follow up to check if the solution I have provided for the issue worked? or let us know if you need any further assistance.
Thanks and regards,
Anjan Kumar Chippa