Forum Discussion
table refresh doesn't find changes but partition refresh does
Hi markmsc ,
Can you please share an update to your issue and let us know if you were able to resolve it by using azure pipeline-orchestrated XMLA refresh of the model, as this seems to be the best possible workaround currently available.
Sharing an update and accepting the reply which may have helped you in the direction will surely help other community memebers who may experience the same error in the future.
Thank you
I tested that this morning and interestingly, no. Here's the metadata for the partition before doing anything:
Here is the metadata after a scheduled refresh of the model run by the service. You've mentioned that a secheduled refresh runs an Automatic. You can see that the bookmark and the last processed datetime for the partition are unchanged from the above after the scheduled refresh:
Next, here is the metadata after then running a manual Full on the table in question. This still did not drop the rows from the partition. Both process datetime and bookmark are again unchanged.
Last, here is a manual Full refresh of the partition. This did drop out the rows as expected. Notice that the last processed datetime has now advanced, but the bookmark still has not changed:
Sounds like you need to dig deeper on the bookmark and make sure the source does have a different date from the bookmark. If it does, then I'm not quite sure what is going on.
Source definitely does have a different, later datetime each norning than the saved bookmark, and yet the detect data changes polling expression on that column finds no changes and does not process the partition. Or at least now it doesn't; before this week, it worked fine. Now the only way to get those changes picked up is to force a full refresh on the partition.
I appreciate you trying to help. I did learn a thing or two from your comments, so I am better off than when I started here, even if my problem is not solved. Thank you.
Just one final check. The changed date is applied to your older data as well right, not just today
The specific sequence is:
- last scheduled refresh of the day runs at 7:30 PM. Presumably the bookmark should reflect this datetime.
- at 5:00 AM the next day, a process runs that identifies the prior day's data that should be dropped from the model, and sets a delete bit on each applicable row, along with the datetime of that update in the row's lastupdated column. This should be after the bookmark from step 1.
- at 6:00 AM that next day, one hour later, the first scheduled refresh of the day runs. Detect Data Change is set to look at the lastupdated column, and the query filter is set to exclude rows marked as deleted. Despite step 2, it finds no changes from its last run the prior evening, in step 1.