Forum Discussion
Helper IVDaily Incremental Refresh with Delta Capture and Reporting
Super UserSome of them only change once a year, some once a month. Some may change daily but those are definitely smaller data sets.That data is unsuitable for incremental refresh. Incremental refresh only works with immutable data.
You will need to implement your own partition management, up to and including periodic full refreshes across all partitions.
Helper IVBased on your original message, I assumed I didn't actually need to use incremental refresh in the Service. Rather, because the records are so small, that continuously appending new files to the folder should be okay because PBI will be able to just look at the MAX file date and the next "max" file date (today v. yesterday) to keep the refreshes tidy.
- lbendlin
Frankly, if your data volume is low, don't bother with Incremental Refresh. Do a brute force flush and fill.
- gemcityzach
Appreciate your patience and time on this 🙂 Yea, brute force refresh is good. Can you think of any way to deal with the problem of a record changing on a given day and then changing again in a few days? Am I simply stuck with this issue in the fantastic code you wrote for me because of the Nyquist/Shannon sampling issue you described?
I'm just wondering if the Last Updated field could be of any help?
- lbendlin
Your only other option would be switch to CDC and event based reporting. In its simplest form that is a reference table with all changes to all fields being recorded one by one. It's a bit more computation, but requires dramatically less storage, and allows you to recreate the state of each transaction for any point in time you choose. You can then also do things like jitter analysis and sankey flow diagrams etc.