The ultimate Fabric, Power BI, SQL, and AI community-led learning event. Save €200 with code FABCOMM.
Get registeredEnhance your career with this limited time 50% discount on Fabric and Power BI exams. Ends September 15. Request your voucher.
Our ERP system continuously creates a .csv file with entries for each user instance, and it's approximately 50 MB each month.
I remmove the file from the ERP system where it is generated, and use Power Query to extract information I'm looking for (this is just a small part of the file) and result in a much smaller and manageable file I use for my dashboard. I will then next month pull out a new .csv file to update the user statistics but I want to avoid reading in all the previous months large .csv files. I want to merge the new processed file with the previously processed file containing previous months' user information.
I know I could use a folde and collect all the files, but after some months this is not so effective when I open the report. How can I most efficiently merge a previously processed file with a new one without having to process all the data to get an updated file with user information? Please see my drawing trying to explain the situation. Thanks
Solved! Go to Solution.
@pnoteng , I think it should be a append not merge
you can use option like this with folder
https://exceleratorbi.com.au/load-the-latest-version-of-a-file-in-power-query/
or check DAX append
https://blog.crossjoin.co.uk/2020/04/13/keep-the-existing-data-in-your-power-bi-dataset-and-add-new-...
https://www.thebiccountant.com/2017/01/11/incremental-load-in-powerbi-using-dax-union/
@pnoteng , I think it should be a append not merge
you can use option like this with folder
https://exceleratorbi.com.au/load-the-latest-version-of-a-file-in-power-query/
or check DAX append
https://blog.crossjoin.co.uk/2020/04/13/keep-the-existing-data-in-your-power-bi-dataset-and-add-new-...
https://www.thebiccountant.com/2017/01/11/incremental-load-in-powerbi-using-dax-union/
User | Count |
---|---|
64 | |
55 | |
54 | |
51 | |
31 |
User | Count |
---|---|
180 | |
88 | |
70 | |
46 | |
43 |