incremental refresh do not store old data

[yanivshe]

Im using incremental refresh on daily basis refresh.

Its important to say that I'm using the web API data source to get the data.

See diagram and explanation below it:

chronological time events

The refresh (star icon) on 11/6 will grab data for whole 10/6 day (207 rows).

The refresh on 12/6 will grab data for whole 11/6 day (218 rows).

The problem: I'm expecting on 13/6 morning to see 425 rows but I only see 11/6 data (218 rows) - Old data is not stored

Here is a summary of the last cuople of days:

Last 3 refresh status

Here is my incremental refresh configuration: 

my incremental configuration

What am I missing here?

Doesnt the store policy should keep old data?

Is incremental refresh supported using web API data source?

Any help on thos will be much appriciated...

[amitchandak]

@yanivshe , Seem like some wrong setting for Incremental, refer if these can help

https://radacad.com/all-you-need-to-know-about-the-incremental-refresh-in-power-bi-load-changes-only
https://thinkaboutit.be/2020/02/how-do-i-implement-an-incremental-refresh-in-power-bi-free-or-pro/
https://blog.crossjoin.co.uk/2020/04/13/keep-the-existing-data-in-your-power-bi-dataset-and-add-new-...
https://www.nabler.com/articles/power-bi-data-refresh-and-scheduling/
https://www.thebiccountant.com/2017/01/11/incremental-load-in-powerbi-using-dax-union/
https://www.fourmoo.com/2020/03/11/how-to-configure-incremental-refreshing-in-power-bi-with-datekey-...

[yanivshe]

Hi @amitchandak 

None of the links answers my problem.

The configurations and steps they do in them are simple and I did the same as they did (RenageStart, RangeEnd, filter by params...)

Any way thanks for the reply.

[lbendlin]

where do you look at the data?  The only way to confirm that incremental refresh works as designed is to look at the partition lists for your dataset, in tools like SSMS or DAX Studio. You will see if the partitions are properly created, how much data rows each of them contains, and when each partition was last written to.

[yanivshe]

Hi @lbendlin 

Im not familliar with the partition list and power query aspects as I'm new to power BI.

I updated my post to hold more data.

Can you please refer me to steps where I can see who to do what you told?

[lbendlin]

You should not ignore the warning about query folding. Since you are using a data source that may or may not support query folding there are many (MANY) things that can go wrong with incremental refresh.  Since you mentioned Power Query - that is one thing. For example I have a CSV file based Incremental refresh running successfully against a Sharepoint folder (based on the Last modified Date of the CSV files) but ONLY after I ripped out nearly all of my M code and replaced it with equivalent code that somehow magically works with incremental refresh while my original code didn't.  "Simple"  things like the difference between Table.ExpandColumn and Table.Combine. Oh, yeah, and Microsoft support told me it cannot be done (I have the ticket to prove that they claimed that)

If you are just starting with Power BI then you should not work on incremental refresh, especially not with non-foldable data sources.