Forum Discussion

pnoteng's avatar
pnoteng
Regular Visitor
3 years ago
Solved

Update a query with a new data set for previous month without need to read all previous month data

Our ERP system continuously creates a .csv file with entries for each user instance, and it's approximately 50 MB each month.
I remmove the file from the ERP system where it is generated, and use Power Query to extract information I'm looking for (this is just a small part of the file) and result in a much smaller and manageable file I use for my dashboard. I will then next month pull out a new .csv file to update the user statistics but I want to avoid reading in all the previous months large .csv files. I want to merge the new processed file with the previously processed file containing previous months' user information.
I know I could use a folde and collect all the files, but after some months this is not so effective when I open the report. How can I most efficiently merge a previously processed file with a new one without having to process all the data to get an updated file with user information? Please see my drawing trying to explain the situation. Thanks