Forum Discussion
Deduplication troubles
Im sure im missing something simple but appricate the help. Im importing a big XML file that has some duplications in it. Changes some info for provacy but the main point is I get duplications on t...
HI Anonymous,
I think they may relate to your source data structure. For this scenario, you can add filter on the category to filter records equal to blank or filter on tracking field if it equal to IP.
Regards,
Xiaoxin Sheng
Your right the source data contains the duplciation but i am pulling to from an API and I have no ability to clean it up before ingesting the data.
I considerd just filtering out field equal to IP however there are plenty of rows that IP is not a duplcate so i need to keep them in the data. I need to find duplicate rows based on IP and ID matching and remove the line which has IP in the tracking column.