Incremental Refresh Bug

[rdini8]

Hi everyone, I have a report that is set up with incremental refresh. I have this problem and I don't know if it's a bug or it is the way it is. Whenever I refresh the dataset for a particular day, any rows for that day after the refresh time won't load in my dataset anymore. But the next day rows will be loaded. For e.g let's say I have 10 rows before 12:00pm of 1st August and I refresh at 12:00pm of 1st August, the 10 rows will be loaded. Let's say another 5 rows are added after 12:00pm of 1st August, these rows won't be loaded anymore when we refresh again. When we refresh on 2nd August, the 2nd August data will loaded along with 10 rows before 12pm of 1st August but not the 5 rows after 12pm of 1st August.

In summary, data won't be loaded after the refresh time for a specific day. Let's say scheduled refresh is set at 12:00 am then data after 12:00am won't be loaded which is basically the entire day.

Is there some setup wrong from my side or is it suppose to be this way?

[Anonymous · Accepted solution]

Hi, @rdini8 

First of all, the problem you encountered is not a bug but a feature of incremental refreshing
First of all, I need to briefly explain the principle of incremental refresh.
Incremental refreshing requires the existence of data/time fields in the data source itself, and the data source supports query folding.
The reason for the need for data/time type field data:
Incremental refresh is based on the data/time column for data partitioning, will only refresh the data in the refresh partition, at the same time will be a piece of the historical partition data display, that is to say, the service will only display the data in the two partitions of the time period.The time period is judged according to the date and time fields set in the rangestart&rangeend parameters.
For example, it is currently 8/2, I set a refresh interval of 3 days, then only these data will be refreshed in the data source.

Here is a description of the following options when setting incremental refreshes

Incremental refresh for semantic models and real-time data in Power BI - Power BI | Microsoft Learn

Once you understand them properly, use them appropriately and you can get the refresh strategy you need.
If you have configured the correct refresh policies, but they are not refreshing your latest data, you can follow the solution proposed in the following document to solve the current problem:

Troubleshoot incremental refresh and real-time data - Power BI | Microsoft Learn

Best Regards

Jianpeng Li

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

[ajohnso2 · Accepted solution]

I believe you are referring to future dated records (Forecasts/Budgets etc)? I.e records with a date after the refresh date (today).

Unfortunately the GUI does not currently allow you to configure capturing any records after the refresh date, however you can forcefully create this behaviour by using advanced api's to ammend the refresh policy "effective date (refresh date)"

See this for example Advanced incremental refresh via API - All 'OTHER'... - Microsoft Fabric Community

[Anonymous · Accepted solution]

Hi, @rdini8 

First of all, the problem you encountered is not a bug but a feature of incremental refreshing
First of all, I need to briefly explain the principle of incremental refresh.
Incremental refreshing requires the existence of data/time fields in the data source itself, and the data source supports query folding.
The reason for the need for data/time type field data:
Incremental refresh is based on the data/time column for data partitioning, will only refresh the data in the refresh partition, at the same time will be a piece of the historical partition data display, that is to say, the service will only display the data in the two partitions of the time period.The time period is judged according to the date and time fields set in the rangestart&rangeend parameters.
For example, it is currently 8/2, I set a refresh interval of 3 days, then only these data will be refreshed in the data source.

Here is a description of the following options when setting incremental refreshes

Incremental refresh for semantic models and real-time data in Power BI - Power BI | Microsoft Learn

Once you understand them properly, use them appropriately and you can get the refresh strategy you need.
If you have configured the correct refresh policies, but they are not refreshing your latest data, you can follow the solution proposed in the following document to solve the current problem:

Troubleshoot incremental refresh and real-time data - Power BI | Microsoft Learn

Best Regards

Jianpeng Li

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.