Incremental refresh based on deltas from API

[Anonymous]

Hi everyone,

I am struggling with the following problem:

There are two API-Endpoints:

1. /items returns the list of all ItemIds. Called with modified_after=YYYYMMDD, it returns a list of all ItemIds that have been modified or newly created after the specified timestamp (e.g. /items?modified_after=YYYYMMDD).

2. /items/<ItemId> returns the information of the actual item.

Now I want to setup a workflow that initially pulls all data (alternatively all data modified since 2010-01-01) and afterwards, dynamically pulls only those items that have been modified. There is no pattern that says that only recently created items will change, most of them are probably less than 3 months old, but other might be up to 2 years old and I need to catch all of them. Furthermore, I do not like to throw away any of those items in the future.

As far as I understood the incremental refresh, such a distinction between "active" (= changes expected) and passive (= static) would be required to create the partitions. Do you know any way how to solve my issue within a PowerBI Dataflow?

Thank you very much in advance!

Edit: The metadata of the items contain a revision information. So it would also be sufficient to just append all changed items because I can filter out obsolete revisions later in the process.

[lbendlin]

Incremental Refresh is the wrong solution to your issue. It expects data to be immutable.

You can still use partitions (based on the item creation date) but you will need to selectively refresh these partitions based on the last modified dates (you can abuse the "Detect Data Changes"  option for that but it is extremely costly).  If your API source is fast, and you have less than millions of rows you may want to just go with the brute force flush and fill.