Forum Discussion
Overwrite already existing rows when refreshing
Anonymous
Hi,
I feel that in this scenario removing duplicates is a better option to go with as compared to merge table, considering the fact the there is a single souce table.
Rgds,
Vivek
PS: It is good that you are asking questions. That is how we all learn :)
Sorry for the late reply. I experimented a little and I realised that what I ask for can be done with the "Remove Duplicates" option. Seems more simple solution than creating 2 separate tables.
However, does that option deletes the rows or just hides them? What do you think?
Thank you
Anonymous
If your question is whether Remove Duplicate will make changes in the source data then the answer is it will hide the rows. "Remove Duplicates" will keep the first row of the duplicates row and removes/hides the rest before loading to Power BI desktop.
Hope it answers your query.
Rgds,
Vivek
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.
- vivran22
My question is if I remove the duplicates, is the size of the data table (or the space it takes) reduced or does it remain the same?
For example if I open a query in power BI with a 10 million row table and then open the same query but now with only 100 rows because the rest are hidden, will they take the same time to load?
Thank you.Anonymous
As per my understanding, Power Query will load the data first and the perform then desired actions. You may improve the efficiency of the entire process by restricting what all tables/queries be refreshed or to be loaded to Power BI data model.
This article may guide you to some extent
Rgds,Vivek
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.Because I don't understand how I could apply the advice given on this article I will stick with the 2 solutions we discussed so far. So the 2 solutions for my problem are:
1) This article that explains how to override data
2) Hiding duplicate rows
The 2nd option is not memory effective.
If I choose to go with the 1st option, how could I create the override table every time?
For example, let's say original data table has data for dates 13th-16th of June.
I hit refresh button and new data comes in, for the dates 14th-17th of June. Now I need to replace the already existing 14th-16th of June on the original data table with the new ones, and then append the data for 17th on the same table.
To do this, as far as the article says, I will have to create an "override" table, with the new data (14th-17th of June).
But the next time I hit the "refresh" button another bunch of new data will come in, for the dates 15th-18th. What happens now with the override data table? I will need to replace all of the dates on the override table (14th-17th) with the new ones (15th-18th). Am I correct? If yes, how could I do that?
Thank you and sorry for the lot of questions but I am new to power BI