Forum Discussion

Anonymous's avatar
Anonymous
Not applicable
7 years ago
Solved

Overwrite already existing rows when refreshing

I have a database with dates that is refreshed every X hours. I want once I hit the refresh button to overwrite the rows that have the same Date with the new data.   Currently, when I hit the the r...
Need Help
  • vivran22's avatar
    vivran22
    7 years ago

    Anonymous 

     

    Hi,

     

    I feel that in this scenario removing duplicates is a better option to go with as compared to merge table, considering the fact the there is a single souce table.

     

    Rgds,

    Vivek

     

    PS: It is good that you are asking questions. That is how we all learn :)

vivran22's avatar
vivran22
Community Champion
7 years ago

Anonymous 

 

Hi,

 

Ususally when you hit refresh, Power Query will perform all those tasks (which includes merge and append) which has been defined in the Edit Query mode. So, your entire process should  be automated will work with the refresh button.

 

Otherwise, you may share the sample files and I would like to recreate it for the demonstration purpose.

 

Rgds,

Vivek

Anonymous's avatar
Anonymous
Not applicable
7 years ago

vivran22

 

Sorry for the late reply. I experimented a little and I realised that what I ask for can be done with the "Remove Duplicates" option. Seems more simple solution than creating 2 separate tables.

However, does that option deletes the rows or just hides them? What do you think?

 

Thank you

  • vivran22's avatar
    vivran22
    Community Champion
    7 years ago

    Anonymous 

     

    If your question is whether Remove Duplicate will make changes in the source data then the answer is it will hide the rows. "Remove Duplicates" will keep the first row of the duplicates row and removes/hides the rest before loading to Power BI desktop.

     

    Hope it answers your query.

     

    Rgds,

    Vivek

     

    If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

    • Anonymous's avatar
      Anonymous
      Not applicable
      7 years ago
      vivran22

      My question is if I remove the duplicates, is the size of the data table (or the space it takes) reduced or does it remain the same?

      For example if I open a query in power BI with a 10 million row table and then open the same query but now with only 100 rows because the rest are hidden, will they take the same time to load?

      Thank you.
      • vivran22's avatar
        vivran22
        Community Champion
        7 years ago

        Anonymous 


        As per my understanding, Power Query will load the data first and the perform then desired actions. You may improve the efficiency of the entire process by restricting what all tables/queries be refreshed or to be loaded to Power BI data model.


        This article may guide you to some extent


        Rgds,

        Vivek


        If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.