Forum Discussion
Power bi
#QuestionForGroup Hi All. I have a dataset of 2.9M rows which upon unpivoting and merging with 1 table becomes some 246M rows. After that I have to remove duplicates from the table(based on Col1, C...
Hi Ayushman1997 -Since your dataset is exploding from 2.9M → 246M rows, the issue likely stems from unpivoting and merging. Try the following tips like:
- Avoid unpivoting in Power Query; try reshaping the data in SQL or DAX.
- Filter and remove duplicates before merging.
- Reduce columns before unpivoting to keep the dataset small.
- Perform joins at the database level if possible.
Hope this helps.
The unpivot and merge doesn't have any problems as I checked without remove duplicates step and all 246M loaded in 25mins. So the problem lies in "remove duplicates" step.