Forum Discussion
Resolver IIncremental refresh with no parameters advice
- Hi there
Even though you set it to only go back one day, because it cannot do query folding it has to read all the data to find the last day.
Incremental refreshing is the right approach but it only works on data sources that can do query folding which is typically database systems
Resolver IThanks Gilbert!
Yes I do want each new days data to be added be my existing data in my model. I'll try yuor suggestion, since I have no defined range, is there a wildcard indicator I can use to grab all?
Terry
Super UserCurrently you would have to use the RangeStart and RangeEnd which have to be DateTime columns.
- tbobolz
Hi Gilbert,
Sorry for the delay, I'm finally back on this project. Maybe I am not fully understanding this process. I've tried to create my incremental refresh and have read through most of the set-up instructions. I'm able to define paramater as you suggested based on a datetime column. When I define the incremental refresh I get a waring about M query folding (As I'll be grabbing daily csv file overwritten daily from a server file on a gateway connection), but I it does allow me to proceed. It seem this set-up is for refreshing past data as it has me setting-up past year paramaters from the paramater column. Is this still what I need to do. I choose to look back 6 years and it seemed to work fine; yet, I'm concerned about the "folding" warning
Ultimately what I want to do a one time load of the past year's data, then I'd like it to load the new data for each new day starting tomorrow and continue loading the previous days new data from then on out.
This is my first time with this type of set-up, thank you for your help and patience with me.
- GilbertQHi there
The reason for the warning is that typically when setting up incremental refreshing it is done where it can query the data source to only bring back the data it needs.
When using a CSV file there is a high possibility that it will have to read all the CSV files to find the files that need to be updated. And by reading all those files it would be quicker to just refresh them all.
By nature Power BI cannot do query folding on CSV files.- tbobolz
Since I do not need it to read the past rows to look for a changes, I just want it to load the new daily data, should I be using a different refresh method?
Thanks
- tbobolz
Sorry, I actually set my "store rows" and "refresh rows" to 1 day.