Forum Discussion
Helper IIBest Practice: Order Data via API that contains an Order Date and a Last Modified Date
but how do I setup an incremental refresh on a value that is changing?What you are looking for is differential refresh, not incremental refresh. Incremental refresh assumes that your data is largely immutable, and it can only work on partition level, not row level.
How you handle that is up to you. Think of it in terms of cold-warm-hot data. You refresh the hot partitions frequently to capture all changes to recently created rows. You refresh the warm partitions occasionally to capture the changes to older rows. You refresh the cold partitions if you know for sure that a row there has changed.
Consider using SCD instead.
Super UserThis is possible but only makes sense if the API source is actually optimized for queries filtering on Last Modified Date. Let's say the source is a SQL server table, and there is no (maintained) index on that column - then it will be a slow as no filter at all.
You will also want to keep the option open to periodically flush and fill in case you missed some updates, or the query bombed halfway through etc.