Forum Discussion
Solved
Need to reduce refresh times...
Hi, I get a 8 files on the hour every hour each day. The files are dropped into a folder (not sharepoint or onedrive). I was pulling from the folder and appending the all files. But that file appen...
Unfortunately, unless your data source supports Incremental Refresh, and CSV files do not, no matter where they are stored, Power Query will process every single record in every single file every single refresh for all queries that get data loaded to the data model. your only hope of optimization is:
- make your queries more efficient. For example, if you are grouping or merging, grouping before merging can help as there is less to merge.
- Move your data into a server DB, like SQL Server via a script. You get two advantages here:
- Query folding means the server will do a lot of the work for you - perhaps all if 100% of your steps fold
- Incremental Refresh becomes possible.
Unfortunately, unless your data source supports Incremental Refresh, and CSV files do not, no matter where they are stored, Power Query will process every single record in every single file every single refresh for all queries that get data loaded to the data model. your only hope of optimization is:
- make your queries more efficient. For example, if you are grouping or merging, grouping before merging can help as there is less to merge.
- Move your data into a server DB, like SQL Server via a script. You get two advantages here:
- Query folding means the server will do a lot of the work for you - perhaps all if 100% of your steps fold
- Incremental Refresh becomes possible.