Forum Discussion
Incremental refresh is not working as expected
- 1 month ago
Hi Mahipal,
One thing I would add is that the second refresh taking roughly the same amount of time does not necessarily mean incremental refresh is not working.
With your current policy, Power BI will refresh the entire last 5 days on every service refresh. Microsoft's incremental refresh documentation states that all rows within the configured refresh period are refreshed each time.
In your screenshot, Detect data changes is not enabled, so Power BI is not trying to determine whether individual daily partitions actually changed before refreshing them.
The first refresh is different because it creates and loads the historical partitions as well as the recent refresh partitions. After that, only the configured refresh window should normally be processed.
So I would verify what Snowflake is actually receiving before concluding that incremental refresh is failing. In Snowflake Query History, compare the SQL generated by the first and second Power BI Service refreshes and check whether the later refresh contains LASTMODIFIEDDATE predicates covering only the recent 5-day window.
If the second refresh is querying only those recent dates, then incremental refresh is working even if the total duration happens to be similar.
If you have another reliable audit/change timestamp, you could also consider Detect data changes. Microsoft documents that this can skip periods whose maximum change value has not changed since the previous refresh. I would not use the same column that you are already using for RangeStart / RangeEnd partitioning.
So I would check:
- whether the RangeStart / RangeEnd filter folds to Snowflake;
- what date predicates appear in Snowflake Query History on the second refresh;
- how much of your total data volume actually falls inside those five days.
That should tell you whether the issue is the incremental-refresh policy or simply that the five-day refresh window itself is still expensive.AI-assisted drafting: AI was used to help structure and phrase this response. I reviewed and validated the technical content before posting.
Hi Mahipal,
To pinpoint why the second refresh isn't faster, check these three specific areas in addition to query folding:
- Snowflake Warehouse State (Cold vs. Warm Cache): If your Snowflake virtual warehouse auto-suspends between refreshes, the second run hits cold local SSD cache and has to spin up compute from scratch, often taking similar time even for fewer rows.
- Clustering & Micro-partition Pruning: Check the Query Profile in Snowflake for the second refresh. If LASTMODIFIEDDATE isn’t part of the clustering key, Snowflake may perform a full table micro-partition scan (partitions scanned equals partitions total) despite the WHERE clause.
- Custom SQL Import: If the query was loaded using custom SQL (Value.NativeQuery) rather than navigating through schema tables, incremental partition filters break folding unless [EnableFolding=true] is explicitly added in M.
If this helped resolve your issue, please mark it as the Accepted Solution and drop a kudos to help others in the community find it!
Thanks
Kannan N