Forum Discussion
Update a query with a new data set for previous month without need to read all previous month data
Our ERP system continuously creates a .csv file with entries for each user instance, and it's approximately 50 MB each month.
I remmove the file from the ERP system where it is generated, and use Power Query to extract information I'm looking for (this is just a small part of the file) and result in a much smaller and manageable file I use for my dashboard. I will then next month pull out a new .csv file to update the user statistics but I want to avoid reading in all the previous months large .csv files. I want to merge the new processed file with the previously processed file containing previous months' user information.
I know I could use a folde and collect all the files, but after some months this is not so effective when I open the report. How can I most efficiently merge a previously processed file with a new one without having to process all the data to get an updated file with user information? Please see my drawing trying to explain the situation. Thanks
pnoteng , I think it should be a append not merge
you can use option like this with folder
https://exceleratorbi.com.au/load-the-latest-version-of-a-file-in-power-query/
or check DAX append
https://blog.crossjoin.co.uk/2020/04/13/keep-the-existing-data-in-your-power-bi-dataset-and-add-new-data-to-it-using-incremental-refresh/
https://www.thebiccountant.com/2017/01/11/incremental-load-in-powerbi-using-dax-union/
1 Reply
- amitchandak
Super User
pnoteng , I think it should be a append not merge
you can use option like this with folder
https://exceleratorbi.com.au/load-the-latest-version-of-a-file-in-power-query/
or check DAX append
https://blog.crossjoin.co.uk/2020/04/13/keep-the-existing-data-in-your-power-bi-dataset-and-add-new-data-to-it-using-incremental-refresh/
https://www.thebiccountant.com/2017/01/11/incremental-load-in-powerbi-using-dax-union/