Hi Keerthi,
I ran into this exact issue and found a workaround that solved it completely.
The fix: set the "Incrementally refresh data starting" window to the same duration and granularity as your archive window ("Store rows in the last"). For example, if you store 10 years of history, set the incremental window to 10 years too — instead of a small window like 30 days.
Detect Data Changes only checks partitions that fall inside the incremental window. If that window is small, historic partitions are simply never checked. Making the window match your full archive means every partition gets checked on every refresh — and thanks to the refresh bookmark, unchanged partitions are still skipped automatically, so you're not paying for a full reload each time, just a lightweight check per partition.
One key adjustment to make this practical: reduce your partition count. With a wide incremental window, every refresh checks every partition, so more partitions means more overhead. Switching from monthly to yearly granularity cut our total partition count by ~10x and dramatically reduced refresh duration.
With this approach, we went from a full refresh to fully automatic incremental detection across a 10-year history, and measured a ~70-76% reduction in Fabric CU consumption compared to the original setup — validated across multiple days, not a one-off measurement.
Happy to share more detail if useful.
