Forum Discussion
incremental refresh do not store old data
Im using incremental refresh on daily basis refresh.
Its important to say that I'm using the web API data source to get the data.
See diagram and explanation below it:
chronological time events
The refresh (star icon) on 11/6 will grab data for whole 10/6 day (207 rows).
The refresh on 12/6 will grab data for whole 11/6 day (218 rows).
The problem: I'm expecting on 13/6 morning to see 425 rows but I only see 11/6 data (218 rows) - Old data is not stored
Here is a summary of the last cuople of days:
Last 3 refresh status
Here is my incremental refresh configuration:
my incremental configuration
What am I missing here?
Doesnt the store policy should keep old data?
Is incremental refresh supported using web API data source?
Any help on thos will be much appriciated...
5 Replies
where do you look at the data? The only way to confirm that incremental refresh works as designed is to look at the partition lists for your dataset, in tools like SSMS or DAX Studio. You will see if the partitions are properly created, how much data rows each of them contains, and when each partition was last written to.
You should not ignore the warning about query folding. Since you are using a data source that may or may not support query folding there are many (MANY) things that can go wrong with incremental refresh. Since you mentioned Power Query - that is one thing. For example I have a CSV file based Incremental refresh running successfully against a Sharepoint folder (based on the Last modified Date of the CSV files) but ONLY after I ripped out nearly all of my M code and replaced it with equivalent code that somehow magically works with incremental refresh while my original code didn't. "Simple" things like the difference between Table.ExpandColumn and Table.Combine. Oh, yeah, and Microsoft support told me it cannot be done (I have the ticket to prove that they claimed that)
If you are just starting with Power BI then you should not work on incremental refresh, especially not with non-foldable data sources.
yanivshe , Seem like some wrong setting for Incremental, refer if these can help
https://radacad.com/all-you-need-to-know-about-the-incremental-refresh-in-power-bi-load-changes-only
https://thinkaboutit.be/2020/02/how-do-i-implement-an-incremental-refresh-in-power-bi-free-or-pro/
https://blog.crossjoin.co.uk/2020/04/13/keep-the-existing-data-in-your-power-bi-dataset-and-add-new-data-to-it-using-incremental-refresh/
https://www.nabler.com/articles/power-bi-data-refresh-and-scheduling/
https://www.thebiccountant.com/2017/01/11/incremental-load-in-powerbi-using-dax-union/
https://www.fourmoo.com/2020/03/11/how-to-configure-incremental-refreshing-in-power-bi-with-datekey-or-date-instead-of-the-default-datetime/Hi amitchandak
None of the links answers my problem.
The configurations and steps they do in them are simple and I did the same as they did (RenageStart, RangeEnd, filter by params...)
Any way thanks for the reply.
