Forum Discussion
Helper Iincremental refresh policy
Helper I
Super UserI think the real question to answer is how many months back can a record be changed after it's created date? That will help determine your incremental refresh time as you will need to include that many months potentially.
lbendlin has great insight in his response that confirms you will need to stick with the created_at for partitioning.
The trick will be finding the right balance of how far back to things get updated and what time frame needs to be incramentally refreshed.
- prasadpatil020
We are currently displaying 6 months of data. However, any order from the past 5 months can still be updated, and we are not certain how far back an order may be updated.
To address this, we are considering removing the partitioning from the GCP backend table itself. This would allow us to retrieve all updated records from the source system without being restricted by the current partitioning strategy. On the Power BI side, we can then implement an incremental refresh policy using the Updated Date column, so that only records with recent updates are refreshed while still ensuring that all historical updates are captured.