Forum Discussion
Incremental Refresh on Dataflow sourced from SharePoint
- 5 months ago
Hi Odiadri
The reason for putting the range start and range end filter early in the steps is that because all of these subsequent steps thereafter will only be using the filtered data and not the entire data set. The reason for the speed increase is because under the hoods the incremental refresh means it does not have to reload the entire data set every day, but only the new data that is incoming and that is why you are seeing the performance gains.
- 5 months ago
Ah, we haven't mentioned caches yet, have we? Most likely all of your files are only read once from source, and then fetched from cache for the other partitions.
Hi Odiadri
I wanted to check if you had the opportunity to review the information provided. Please feel free to contact us if you have any further questions.
Thank you.