Forum Discussion

gemcityzach's avatar
gemcityzach
Icon for Helper IV rankHelper IV
2 years ago
Solved

Daily Incremental Refresh with Delta Capture and Reporting

Hey folks, I need to design a process that stages and appends a daily CSV file that is a snapshot of a table in a database (no direct connection allowed). This process needs to identify if any recor...
Need Help
lbendlin's avatar
lbendlin
Icon for Super User rankSuper User
2 years ago

Frankly, if your data volume is low, don't bother with Incremental Refresh.  Do a brute force flush and fill.

gemcityzach's avatar
gemcityzach
Icon for Helper IV rankHelper IV
2 years ago

Appreciate your patience and time on this 🙂 Yea, brute force refresh is good. Can you think of any way to deal with the problem of a record changing on a given day and then changing again in a few days? Am I simply stuck with this issue in the fantastic code you wrote for me because of the Nyquist/Shannon sampling issue you described?

 

I'm just wondering if the Last Updated field could be of any help? 

  • lbendlin's avatar
    lbendlin
    Icon for Super User rankSuper User
    2 years ago

    Your only other option would be switch to CDC and event based reporting. In its simplest form that is a reference table with all changes to all fields being recorded one by one.  It's a bit more computation, but requires dramatically less storage, and allows you to recreate the state of each transaction for any point in time you choose. You can then also do things like jitter analysis and sankey flow diagrams etc.