Forum Discussion
Understanding Incremental Refresh
I think this is due to the size of the query and the request timing outI don't think so. That would have resulted in a different error message. Might have been a transient network connectivity issue, or someone restarting the service on the other end.
What I am looking to do is to refresh data only where a value in a column has changed to cut down on the size of the output handled by scheduled refresh. would incremental refresh help with this?yes and no. While incremental refresh offers to check for data changes this comes at a horrible cost - a canary dataset copy will be created that then does the delta tracking.
Fundamentally incremental refresh expects immutable data (like IoT sensor readings). It is very bad at real world business data (which would require differential refresh). You can mitigate that by keeping your partitions small and by refreshing individual partitions as needed, and doing a full refresh every now and then.
I think this is due to the size of the query and the request timing outI don't think so. That would have resulted in a different error message. Might have been a transient network connectivity issue, or someone restarting the service on the other end.
What I am looking to do is to refresh data only where a value in a column has changed to cut down on the size of the output handled by scheduled refresh. would incremental refresh help with this? yes and no. While incremental refresh offers to check for data changes this comes at a horrible cost - a canary dataset copy will be created that then does the delta tracking.
Fundamentally incremental refresh expects immutable data (like IoT sensor readings). It is very bad at real world business data (which would require differential refresh). You can mitigate that by keeping your partitions small and by refreshing individual partitions as needed, and doing a full refresh every now and then.
Hi,
Thanks for your help. Figured out the issue... sort of. We ended up caching the results of the query and then trying cloud refresh... and viola, it works. So this error had to do with the lag time between getting the results of the query and sending the request.
TY