Forum Discussion
Incremental Refresh isn't dropping old data
AlexisOlson Yes, I know how incremental is "supposed" to work.
Actually this "incremental" refresh does accomplish what i want. It doesn't refresh the entire dataset the entire time. Just the days that have been updated.
My question is, why isn't it dropping data? The dataset grows when it shouldn't.
This error makes me think my other incremental updates (that actually are setup to store historical data) aren't dropping data either and growing more than they should as well.
It does seem like it should be dropping the older data. Maybe it has to do with the way it partitions the data? If the data you want to drop is in the same partition as some of the data you still want, maybe it just keeps the whole partition.
This text from the documentation I linked previously makes me think it's preserving more than strictly the period you specify:
The Archive data starting before refresh date setting determines the historical period in which rows with a date/time in that period are included in the model, plus rows for the current incomplete historical period, plus rows in the refresh period up to the current date and time.
For example, if you specify five years, the table stores the last five whole years of historical data in year partitions. The table will also include rows for the current year in quarter, month, or day partitions, up to and including the refresh period.