Incremental Refresh - Keep yesterday's data and refresh today's.

[kasiaw29]

Hi all, 

I'm setting up incremental refresh for the first time with one of my reports. I'm most likely doing something silly but the main goal here is to load the data in and only refresh the next day's data, i.e. if I load my data in I want to keep it all and perform a refresh tomorrow to bring in new data. I thought I had it set up right in the service but I checked my report and it's empty. 

I've got my custom filter set up as such

and incremental refresh as such 

I see that it says data will be incrementally refreshed from 17/08/2022 to 17/08/2022 which means yesterday and override yesterday's load which I don't want to do. 

Data in the table changes every day and what I'm trying to achieve with this is to capture daily changes in data via incremental refresh to perform trend analysis of it. 

[lbendlin · Accepted solution]

Data in the table changes every day

That means your data is unsuitable for incremental refresh.  Incremental refresh expects immutable data that can be added on to, but never needs to be updated.

You are looking for differential refresh. That is not something Power BI can do.

Your best bet is to extend the refresh window to refresh not the last day but the last 14 days (for example) to catch most changes, and to run occasional full refreshes to cover cases when older data has changed.

Note that you can also manually or programmatically refresh individual partitions.

Note that you have selected "Only refresh complete days" which is most likely not what you want.

[Martin1986]

Late reply but for those that come here after a search:

I believe it needs an immutable datetime column, not necessarily that the whole row needs to be immutable. If you have an immutable column like a creationdate log column you can use that together with detect data changes on a modifieddate log column. If you set your partitions at the month level for example it will scan each month for a change in last modifieddate and then refresh the entire partition. 

[kasiaw29]

Thank you for your reply!