Forum Discussion
Helper IVDaily Incremental Refresh with Delta Capture and Reporting
Super UserThat seems like a manageable amount of data. I would just ingest the CSVs as is. If you have the file datetime inside the data then that is ok too, but the filename datetime is usually enough.
For the delta processing you can use the standard period over period patterns but also have a look at EXCEPT and INTERSECT.
Helper IVDoes it matter if my data does not have an audit date time field? Unfortuantely, it doesn't look like any of the target tables have a "lastUpdated" or "LastModified" field exposed to the report layer export. So the only thing I have are exact snapshots each day of the same fields with a file name of the previous day's export and an appended 'snapshot date' field into each record of the target snapshot.
- lbendlin
the file date is sufficient.
- gemcityzach
Thank you. I am still kind of lost. I've read the EXCEPT, INTERSECT and a bunch of tutorials about combining two tables. But my problem is that I don't have tables or distinct queries, I have one query with appended data already in it. I.e., my single query has 03-01-2024 through 03-31-2024 data all essentially stacked on top of itself with a 'snapshot date' field.
It's escaping me how on a rolling daily basis I compare today's records to yesterday's records to identify any records that have changed since I don't have an audit field (lastModified).
- lbendlin
give me some sample data to work with and I can show you the mechanics. "Table" can be a physical thing, but it can also be your filter context, or a table variable.