Forum Discussion
Help needed: Detect Data Changes on Modified column with Incremental Refresh
Hi Natarajan_M ,
Thank you for your detailed response to my question I really appreciate it.
I’ve confirmed that the Modified date is stored as a DATETIME field.
Questions:
I'm currently configured for "last 2 days data refresh" while keeping 5 years of historical data.
If I change the historical data range to 60 months, should I also adjust the "last 2 days refresh" value accordingly?As shown in the screenshots attached to my original post, does the "last 2 days refresh only" setting explain why 2021 records with updated Modified dates are not being picked up by Detect Data Changes?
Hi automation , The "Detect Data Changes" feature operates within the refresh context window. It retains historical data for up to 60 months and allows for incremental data refresh for the same duration. With this setting, all your partitions fall within the "Detect Data Changes" window, enabling the system to identify changes in the data. Consequently, not all partitions will be processed until a change occurs within those partitions.
thanks ,
If this response was helpful in any way, I’d gladly accept a kudo.
Please mark it as the correct solution. It helps other community members find their way faster
Hi Natarajan_M ,
I've implemented a last 60 months incremental refresh with data detect changes on my dataset. However, with Power BI Premium Per User (PPU) license, the dataset is not refreshing completely and I'm encountering timeout errors.
Could you please help me resolve this issue?
Thanks!
Hi automation,
This behavior is expected when using a large incremental refresh window with Detect data changes on a Power BI Premium Per User (PPU) license. With a last 60 months refresh policy enabled, the service must evaluate every partition within that window during each refresh to check for changes. Even though Detect data changes helps avoid unnecessary reloads, Power BI still issues polling queries per partition, and if many partitions are involved, this can quickly exceed the execution and resource limits of PPU, resulting in refresh timeouts.
It’s also important to note that Detect data changes operates at the partition level, not at the individual row level. If a single row changes within a large partition, the entire partition needs to be reprocessed, which further increases refresh cost and duration. This becomes especially noticeable when partitions span long periods such as months or years.
To mitigate this, you can try reducing the incremental refresh window (for example, from 60 months to a smaller range that aligns with how often historical data actually changes), and ensure that the column used for RangeStart/RangeEnd and Detect data changes supports full query folding back to the source. Proper partitioning and folding are critical to keeping refreshes efficient.
Thanks,
prashanth
Hi automation ,
After you publish the PBIX file, the first load will involve a complete refresh, meaning that all partitions will be processed. Please verify whether the size of the semantic model exceeds the capacity limits.
To overcome the size issue you can follow the below approach :define a parameter as LoadAllData
In Power Query stepLet .... SampleData = Table.FirstN(Source, 10), Check = if LoadAllData then Source else SampleData in Check
Keep the default value as False and publish the model to service and do the first refresh .
The first refresh will take care of creating all the partitions in the servivce , now set the LoadAllData to True and process the partitions one by one using ssms or fabric notebook.Thanks
If this response was helpful in any way, I’d gladly accept a kudo.
Please mark it as the correct solution. It helps other community members find their way faster