Append with only new rows of data

[monojchakrab]

I am running into a problem while appending new data to an existing query.

I have a query which has got data till ytd Apr'22. But when I get new data every week for Apr and try to append it to the existing query (#YtdApr22), it duplicates the rows of data for the month of apr.

E.g. if #ytdapr22 had data upto 8th of Apr'22 and now i have a new set of data for mtd 16th Apr'22, if I append this new table to the existing table, it duplicates the rows from 1-8th apr already in the existing table@YtdApr22.

Is there a smarter solution at hand other than removing duplicates or deleting the Apr rows from the existing table every time I have to append and refresh the data with the new dates?

[vojtechsima · Accepted solution]

Hi, @monojchakrab 
Isn'T that correct behaviour?
If you have data from 2022/01/01 till 2022/04/08 and you then append new data that are from 2022/04/01 till 2022/04/16, you will append the duplicates too.
If you get your update by week but it contains history of month, try filtering just the current week from the Updat table and then append it.
Or just simply remove duplicates automatically after the append, there's no manual work then after refresh.

[vojtechsima · Accepted solution]

Hi, @monojchakrab 
Isn'T that correct behaviour?
If you have data from 2022/01/01 till 2022/04/08 and you then append new data that are from 2022/04/01 till 2022/04/16, you will append the duplicates too.
If you get your update by week but it contains history of month, try filtering just the current week from the Updat table and then append it.
Or just simply remove duplicates automatically after the append, there's no manual work then after refresh.

[monojchakrab]

yes I could do that...thanks for pointing me in the right direction @vojtechsima . 

So you are saying I take the new MTD table every week and create a filtered table from that to keep only the most recent week and then append to the existing #Ytd table?

[vojtechsima · Accepted solution]

@monojchakrab 
You can do that, 
Or you can just filter the Update table so it contains just the current week (if your column is a Date column, there should be a dynamic function that lets you keep just the current week), if you don'T need to append historical data for the whole month, which I suppose you don't.