Forum Discussion
Combining new data into existing data daily
Hi all,
I hope someone is able to offer me some advice with a problem i've been trying to figure out for a while now.
So I have 2 data sets with the same columns, one is static (data set 1) and the other refreshes daily (data set 2). Sometimes data set 2 will contain duplicates of what is in data set 1, but sometimes there is new data that I then need to be added to data set 1 (the duplicate data can be discarded).
So then 2 things:
1 - How do I set it up so that data set 2 combines with data set 1 on a daily basis removing the duplicates and adding the new data?
2 - How do I date stamp the new data? I want to include within data set 1 the date that the new data from data set 2 is added to that report.
Any help will be appreciated.
Regards,
Emily.
HI EmilyM2019,
#1, You can add a custom step in the dynamic query table with Table.Combine and Table.Distinct functions to combine and remove duplicate records:
Custom= Table.Distinct(Table.Combine{Query1,#"Previous Step"}))#2, Nope, query tables are the processing review of current query steps. It wasn't able to save/log the specific operation timestamps in the current query table fields. I'd like to suggest you remark the operation timestamp on your Datasource side.
Regards,
Xiaoxin Sheng
1 Reply
HI EmilyM2019,
#1, You can add a custom step in the dynamic query table with Table.Combine and Table.Distinct functions to combine and remove duplicate records:
Custom= Table.Distinct(Table.Combine{Query1,#"Previous Step"}))#2, Nope, query tables are the processing review of current query steps. It wasn't able to save/log the specific operation timestamps in the current query table fields. I'd like to suggest you remark the operation timestamp on your Datasource side.
Regards,
Xiaoxin Sheng