Forum Discussion
Helper IIBest Practice: Order Data via API that contains an Order Date and a Last Modified Date
but how do I setup an incremental refresh on a value that is changing?What you are looking for is differential refresh, not incremental refresh. Incremental refresh assumes that your data is largely immutable, and it can only work on partition level, not row level.
How you handle that is up to you. Think of it in terms of cold-warm-hot data. You refresh the hot partitions frequently to capture all changes to recently created rows. You refresh the warm partitions occasionally to capture the changes to older rows. You refresh the cold partitions if you know for sure that a row there has changed.
Consider using SCD instead.
Helper IIThanks lbendlin ,
It's indexed, so no issue there.
Big picture, how do I do that? I get how to setup an incremental refresh based on the order data, because that is going to be fixed, but how do I setup an incremental refresh on a value that is changing? Or am I not actually doing an incremental refresh? Do I not then run the risk of introducing duplicate orders everytime the order updates?
Easiest way to setup a flush and fill? Just via desktop and manually updating that Last Modified value?
- lbendlin
Super User
but how do I setup an incremental refresh on a value that is changing?What you are looking for is differential refresh, not incremental refresh. Incremental refresh assumes that your data is largely immutable, and it can only work on partition level, not row level.
How you handle that is up to you. Think of it in terms of cold-warm-hot data. You refresh the hot partitions frequently to capture all changes to recently created rows. You refresh the warm partitions occasionally to capture the changes to older rows. You refresh the cold partitions if you know for sure that a row there has changed.
Consider using SCD instead.