Forum Discussion
Dataflow Scheduled Refresh Doesn't Load Recent Data
We had a similar issue with incremental refresh recently.
The table had a policy to store rows from the past 1 year and refresh the past 1 month of data. We also had the Detect data changed box ticked and using an Amended_DateTime field to detect data changes.
We were using a Created_DateTime field to filter by and it was thought this would be sufficient to detect new records on its own. For a newly created record, the software would set the Amended_DateTime to null logically because the record hadn't yet been changed.
What we found in test was, the initial load of some sample data was fine as was the first subsequent refresh but further refreshes would not recognise new rows until the record was changed and Amended_DateTime was set to a non null value. In our case, this was happening several days later on the live system.
The solution was to change the source program so that when a new record is created then both the Created_DateTime and Amended_DateTime fields are set to the same value.
We had a similar issue with incremental refresh recently.
The table had a policy to store rows from the past 1 year and refresh the past 1 month of data. We also had the Detect data changed box ticked and using an Amended_DateTime field to detect data changes.
We were using a Created_DateTime field to filter by and it was thought this would be sufficient to detect new records on its own. For a newly created record, the software would set the Amended_DateTime to null logically because the record hadn't yet been changed.
What we found in test was, the initial load of some sample data was fine as was the first subsequent refresh but further refreshes would not recognise new rows until the record was changed and Amended_DateTime was set to a non null value. In our case, this was happening several days later on the live system.
The solution was to change the source program so that when a new record is created then both the Created_DateTime and Amended_DateTime fields are set to the same value.