Forum Discussion
Helper IIncremental refresh is not working as expected
Hi Mahipal,
One thing I would add is that the second refresh taking roughly the same amount of time does not necessarily mean incremental refresh is not working.
With your current policy, Power BI will refresh the entire last 5 days on every service refresh. Microsoft's incremental refresh documentation states that all rows within the configured refresh period are refreshed each time.
In your screenshot, Detect data changes is not enabled, so Power BI is not trying to determine whether individual daily partitions actually changed before refreshing them.
The first refresh is different because it creates and loads the historical partitions as well as the recent refresh partitions. After that, only the configured refresh window should normally be processed.
So I would verify what Snowflake is actually receiving before concluding that incremental refresh is failing. In Snowflake Query History, compare the SQL generated by the first and second Power BI Service refreshes and check whether the later refresh contains LASTMODIFIEDDATE predicates covering only the recent 5-day window.
If the second refresh is querying only those recent dates, then incremental refresh is working even if the total duration happens to be similar.
If you have another reliable audit/change timestamp, you could also consider Detect data changes. Microsoft documents that this can skip periods whose maximum change value has not changed since the previous refresh. I would not use the same column that you are already using for RangeStart / RangeEnd partitioning.
So I would check:
- whether the RangeStart / RangeEnd filter folds to Snowflake;
- what date predicates appear in Snowflake Query History on the second refresh;
- how much of your total data volume actually falls inside those five days.
That should tell you whether the issue is the incremental-refresh policy or simply that the five-day refresh window itself is still expensive.AI-assisted drafting: AI was used to help structure and phrase this response. I reviewed and validated the technical content before posting.
Community ChampionHi Mahipal,
One thing I would add is that the second refresh taking roughly the same amount of time does not necessarily mean incremental refresh is not working.
With your current policy, Power BI will refresh the entire last 5 days on every service refresh. Microsoft's incremental refresh documentation states that all rows within the configured refresh period are refreshed each time.
In your screenshot, Detect data changes is not enabled, so Power BI is not trying to determine whether individual daily partitions actually changed before refreshing them.
The first refresh is different because it creates and loads the historical partitions as well as the recent refresh partitions. After that, only the configured refresh window should normally be processed.
So I would verify what Snowflake is actually receiving before concluding that incremental refresh is failing. In Snowflake Query History, compare the SQL generated by the first and second Power BI Service refreshes and check whether the later refresh contains LASTMODIFIEDDATE predicates covering only the recent 5-day window.
If the second refresh is querying only those recent dates, then incremental refresh is working even if the total duration happens to be similar.
If you have another reliable audit/change timestamp, you could also consider Detect data changes. Microsoft documents that this can skip periods whose maximum change value has not changed since the previous refresh. I would not use the same column that you are already using for RangeStart / RangeEnd partitioning.
So I would check:
- whether the RangeStart / RangeEnd filter folds to Snowflake;
- what date predicates appear in Snowflake Query History on the second refresh;
- how much of your total data volume actually falls inside those five days.
That should tell you whether the issue is the incremental-refresh policy or simply that the five-day refresh window itself is still expensive.
AI-assisted drafting: AI was used to help structure and phrase this response. I reviewed and validated the technical content before posting.