Supplies are limited. Contact info@espc.tech right away to save your spot before the conference sells out.
Get your discountScore big with last-minute savings on the final tickets to FabCon Vienna. Secure your discount
Our ERP system continuously creates a .csv file with entries for each user instance, and it's approximately 50 MB each month.
I remmove the file from the ERP system where it is generated, and use Power Query to extract information I'm looking for (this is just a small part of the file) and result in a much smaller and manageable file I use for my dashboard. I will then next month pull out a new .csv file to update the user statistics but I want to avoid reading in all the previous months large .csv files. I want to merge the new processed file with the previously processed file containing previous months' user information.
I know I could use a folde and collect all the files, but after some months this is not so effective when I open the report. How can I most efficiently merge a previously processed file with a new one without having to process all the data to get an updated file with user information? Please see my drawing trying to explain the situation. Thanks
Solved! Go to Solution.
@pnoteng , I think it should be a append not merge
you can use option like this with folder
https://exceleratorbi.com.au/load-the-latest-version-of-a-file-in-power-query/
or check DAX append
https://blog.crossjoin.co.uk/2020/04/13/keep-the-existing-data-in-your-power-bi-dataset-and-add-new-...
https://www.thebiccountant.com/2017/01/11/incremental-load-in-powerbi-using-dax-union/
@pnoteng , I think it should be a append not merge
you can use option like this with folder
https://exceleratorbi.com.au/load-the-latest-version-of-a-file-in-power-query/
or check DAX append
https://blog.crossjoin.co.uk/2020/04/13/keep-the-existing-data-in-your-power-bi-dataset-and-add-new-...
https://www.thebiccountant.com/2017/01/11/incremental-load-in-powerbi-using-dax-union/