Forum Discussion
Helper Iincremental refresh policy
Helper IHi Community..!!
My data is sourced from GCP, where the underlying table is partitioned on the created_at column. In Power BI, I have configured Detect Data Changes using the updated_at column.
Our requirement is to retain all historical data while also ensuring that records updated after their initial creation are refreshed and reflected in the report. However, since the source table is partitioned by created_at, updates to records belonging to older partitions may not be picked up during incremental refresh, even when their updated_at value changes.
In summary, we need a solution that allows us to:
- Retain all historical data.
- Refresh only records that have been updated recently.
- Avoid reloading the entire dataset during each refresh.
The challenge appears to be that the source data is partitioned on created_at, while the refresh requirement is driven by updated_at.
Super User"The challenge appears to be that the source data is partitioned on created_at, while the refresh requirement is driven by updated_at."
Partition boundaries must be set by immutable fields, so created_at is exactly the right field for that, and "updated_at" is exactly the wrong field.
Remember that "Detect Data changes" will reprocess the ENTIRE partition, not just the changed records.