Forum Discussion
Table Merge creates duplicate records
The problem lay with the Data.
The LEFT JOIN functionality of PowerBI works perfectly.
Hi people.
I've been looking for this issue with no luck as well until I founded a valid work around.
the work around goes like this ::
-Merge your sheets as you like (you don't have to merge on more than one column, one is fine).
-Add index column
-Expand your merged column (this will create duplicate values, the main issue we're looking into).
-since the values in the "index column" were purely unique, now Remove duplicates using index column.
-and now your data is back to original as it was inteded to be.
othmanalazzam wrote:Hi people.
I've been looking for this issue with no luck as well until I founded a valid work around.
the work around goes like this ::
-Merge your sheets as you like (you don't have to merge on more than one column, one is fine).
-Add index column
-Expand your merged column (this will create duplicate values, the main issue we're looking into).
-since the values in the "index column" were purely unique, now Remove duplicates using index column.
-and now your data is back to original as it was inteded to be.
My duplicate issue is solved by applying this method. Thank you very much.
I've been working with a sales table and trying to create dimension tables from the table with all of the fields. I've been frustrated, because I could not tell where the duplicate rows of data were coming from; however, could tell they happened after one of the two merge query steps.
I can affirm that having a unique index prior to the merge, and then removing duplicates from this field works.
Does anyone know why this occurs?
this approach should be accepted as solution. It perfectly resolved the issue.
Thanks! This solution works like a charm
Don't know why Microsoft don want to fix this simple issue....Thanks. This solved my issue.
I work with very extensive datas sheets and as soon as I tried this solution (that works pretty well) the uptade started to take hours to be conclude, thats the only issue Im having right now... Any clues about that?
Having the same issue. Performance severely degrades when I use this method. It works in theory, but in some cases will take forever.
hello there,
the slowness is not coming from adding an index column or removing the duplicate, Power BI and Excel Power Queries are always having a problem when using the "Merge" method, this is happening because "merge" includes a "reference" step inside of it, this specific step "reference" is a performance killer, it really slows everything down in a horrible way, the only two solutions I found to this problem is either you find a workaround to this and try to achieve your result without using "merge" or "reference" (a workaround that does the same effect and I have done that sometimes), the other solution is to not use it all together, there is really no way to optimize the "merge" itself.Regards,
hopefully you'll reply but when I do this ...it does indeed eliminate the duplicates in the 2nd table added
but after removing the dups in the index, now the original table has dups
here is sample data ..trying to merge "ycr" and "integrated"and end up with "goal"
