Forum Discussion
Solved
Incremental Refresh with Elastic Tables and Data Detection
I have a scenario where I am trying to implement incremental refresh on an elastic table using the powerbi service and a semantic model. I'm using the ExecuteCosmosSqlQuery endpoint, which allows me...
Hi jdc2019,
Thank you for your follow-up! You’ve raised an excellent question regarding how Power BI’s incremental refresh engine determines partition updates.
Yes, the Power BI refresh engine evaluates partitions using the modifiedon field, but for it to work optimally, the query must still be folding at the point where modifiedon is treated as a DateTime.
- The refresh engine assesses partitions using the RangeStart and RangeEnd parameters.
- If Power BI can apply the modifiedon filter while query folding is still intact, it will evaluate which partitions need refreshing based on whether the modifiedon value is greater than the last refresh timestamp.
- If query folding breaks before modifiedon is converted to a DateTime, Power BI cannot efficiently determine changes at the partition level, which may lead to unexpected behavior like full partition replacement.
I trust this information proves useful. If it does, kindly Accept it as a solution and give it a 'Kudos' to help others locate it easily.
Thank you.
lbendlin
Super User
Super Userbut if I run the refresh a second time, all of the records that filter out by the modified on date disappear - the service is not retaining previously loaded records in the incremental refresh period.
The Change detection impacts the entire partition. Power BI Incremental Refresh is incapable of replacing/updating individual rows. So most likely your Last Modified Date is impacting more than just the "hot" partition(s)