Forum Discussion
Append query issues with new data and deleting older data (possible M coding needed)
You've asked several questions so let me see if I can tackle them one at a time:
Question 1) how do we append new data to existing query and delete older dates data?
Without having seen your data, I would take this approach. You are importing files, so just select the most recent 3/4 days of files. Do this:
- Assume files that are like the following in a folder, and I want the most recent 3 files. This could be a hard drive folder, onedrive, or SharePoint folder. The SharePoint method is slighty different, but same concept.
- In Power Query, get data from the folder, and hit Transform
- We want to get the 3 most recently added files. It will be in the Date Modified column of Power Query
- Filter by the latest in there. This is wrong, but will generate the code we want to tweak.
- The formula bar now has:
- Table.SelectRows(Source, let latest = List.Max(Source[Date modified]) in each [Date modified] = latest)
- Change this to
- Table.SelectRows(Source, let latest = List.Min(List.MaxN(Source[Date modified],3)) in each [Date modified] >= latest)
- I changed the List.Max to ListMaxN, and told it to get the top 3. So it has all of the May 3 dates, my top 3 dates. Then I used List.Min() around that to get the smallest value, then i change the = latest to >= latest. Now it will always pull the latest 3 files, unless there is some sort of tie where more than one file as identical date/time stamps.
- Press the Combine button, and it will walk you through a wizard to combine the files.
- You'll notice in your queries on the left Power Query created a folder of queries and other folders. This may be the folder query Greg_Deckler was talking about. It is some magic farie dust Power Query uses when combining files. 😉
Question 2) while appending two tables of different structures, how does one ensure additional columns are also added?
This should happen automatically when you import using the Combine operation above. To validate this, I went into one of the files and added a [Marketing Junk] column, saved it, then went back into Power Query and simply told it to refresh. This was the result:
For the Bike file, the column with the 'x' char I put in there for testing is there. For the Clothing and other files, the column is there, but null.
At this point, if you want to further transform by removing columns, I strongly recommend you select the columns you want and then use the Remove Other Columns functionality. The reason is, if you have a column today called [ID] and you are removing it, if next week the [ID] column is gone from your last 3 files, Power Query will break because there is no [ID] column to remove. but if you select all but the [ID] column and Remove Other Columns, Power Query uses a Table.SelectColumns() function and is really selecting what you want. It doesn't care that [ID] is there and gone.
The flip side of this is if you get a new [Valid Data] column in a new file, that will actually not show up either. It will be in Power Query, but Power Query will discard it.
In that case, it gets tricky. If your new column names follow a predictable pattern (variable1, variable2, variable3) then we can wild-card the Table.SelectColumns().
But that is getting more complex. See how this works for you right now, and let me know if I understood your questions properly.
- Anonymous6 years agoNot applicable
Thank you so much for the detailed response. The 2nd question is answered but i would like to explain my 1st question again.
Problem statement is not to consider the files which are the most recent. In my current situation, each file contains data for the last 4 days. So everyday I get a rolling 4 day report. for instance, today's report (5th May) will consists of files from 1st May to 4th May. For me to maintain a continuous updated dataset (file name = Manual_dataset), i have to manually copy the rows between 1st May to 4th May in a seperate sheet. This seperate file consists of the latest data till today. So tomorrow (6th May) when I receive the next report, I now have updated data from 2nd to the 5th May. Now I have to open the Manual_dataset and then perform these steps
i) delete existing data from 2nd to 4th May
ii) copy data from 2nd to 5th May from the latest report
I followed your instructions and encountered the following error.
As you will see, for a single date (1st May), there are 3 files that are being reference. It is desired to remove data from the older files and only maintain the one from 4th May.
I hope I have explained my question better. Do let me know if my query is within the realm of being achieved within the free version of PowerBi desktop.
- edhans6 years ago
Community Champion
Hi Anonymous
I'm still not sure I fully understand your problem, but here is the approach I would take.
- Store all files in a single folder.
- Have Power Query grab the latest file, and all data.
- Have Power Query grab the 2nd latest file, and then with an Anti-Merge, keep all data for dates that are not in the file in step 2 above.
- Have Power Query grab the 3rd latest file, and then with an Anti-Merge, keep all data for dates that are not in the files in steps 2 and 3 above.
- repeat step 4 until you have all the data you need.
If I'm still not understanding, let me know.
If you need help with implementing the above logic, I'll need sample files to mess with. See links below to facilitate that.
How to get good help fast. Help us help you.
How to Get Your Question Answered Quickly
How to provide sample data in the Power BI Forum- Anonymous6 years agoNot applicable
thank you for the response. however, i do not think this will work since this process has to be done on a daily basis. I am trying to look for ways to make it as less manual as possible.