Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

Score big with last-minute savings on the final tickets to FabCon Vienna. Secure your discount

Reply
pnoteng
Regular Visitor

Update a query with a new data set for previous month without need to read all previous month data

Our ERP system continuously creates a .csv file with entries for each user instance, and it's approximately 50 MB each month.
I remmove the file from the ERP system where it is generated, and use Power Query to extract information I'm looking for (this is just a small part of the file) and result in a much smaller and manageable file I use for my dashboard. I will then next month pull out a new .csv file to update the user statistics but I want to avoid reading in all the previous months large .csv files. I want to merge the new processed file with the previously processed file containing previous months' user information.
I know I could use a folde and collect all the files, but after some months this is not so effective when I open the report. How can I most efficiently merge a previously processed file with a new one without having to process all the data to get an updated file with user information? Please see my drawing trying to explain the situation. Thanks  Annotation 2023-02-17 155047.jpg

1 ACCEPTED SOLUTION
amitchandak
Super User
Super User

@pnoteng , I think it should be a append not merge

 

you can use option like this with folder

https://exceleratorbi.com.au/load-the-latest-version-of-a-file-in-power-query/

 

 

or check DAX append

https://blog.crossjoin.co.uk/2020/04/13/keep-the-existing-data-in-your-power-bi-dataset-and-add-new-...
https://www.thebiccountant.com/2017/01/11/incremental-load-in-powerbi-using-dax-union/

Share with Power BI Enthusiasts: Full Power BI Video (20 Hours) YouTube
Microsoft Fabric Series 60+ Videos YouTube
Microsoft Fabric Hindi End to End YouTube

View solution in original post

1 REPLY 1
amitchandak
Super User
Super User

@pnoteng , I think it should be a append not merge

 

you can use option like this with folder

https://exceleratorbi.com.au/load-the-latest-version-of-a-file-in-power-query/

 

 

or check DAX append

https://blog.crossjoin.co.uk/2020/04/13/keep-the-existing-data-in-your-power-bi-dataset-and-add-new-...
https://www.thebiccountant.com/2017/01/11/incremental-load-in-powerbi-using-dax-union/

Share with Power BI Enthusiasts: Full Power BI Video (20 Hours) YouTube
Microsoft Fabric Series 60+ Videos YouTube
Microsoft Fabric Hindi End to End YouTube

Helpful resources

Announcements
August Power BI Update Carousel

Power BI Monthly Update - August 2025

Check out the August 2025 Power BI update to learn about new features.

August 2025 community update carousel

Fabric Community Update - August 2025

Find out what's new and trending in the Fabric community.

Top Solution Authors