Dataset refresh duration increase - query caching?

We're encountering a very specific issue with one of our main datasets, and I suspect it might have to do with query caching, but want to see what others think. When the dataset is published with a new change, it takes around 10-15 minutes to refresh. This relatively quick refresh duration lasts for a while, but it slowly increases until the next time we publish a new change.

Here is a screenshot of the latest refreshes for reference:

You can see that I published a change yesterday, and now it takes about 12 minutes to refresh, but prior to that it was taking over an hour. The last change prior to yesterday's was published at the start of November. This slow increase has happened on this dataset at least 3 times now, but the rest of our datasets are very consistent.

This is the only dataset with query caching enabled. Could that be the cause of the increased refresh durations? It seems like quite an extreme increase, while the documentation mentions "some performance degradation may occur". I'd appreciate any thoughts on this or how to clear the cache besides republishing the dataset.