Forum Discussion

Anonymous's avatar
Anonymous
Not applicable
2 years ago
Solved

Generating a filter to remove duplicates

Hello,

In my data source, there are three tables: Appeared, Disappeared and Component. I need to remove the duplicates if all of these three tables are the same for different rows. But I also keep adding data everyday, so I need a function/code that will automatically remove duplicates as and when I add more data.

I have combined the three columns into one and called it key. But even if I just try to remove duplicates from the key (transform data), it does not work. Can someone please help me with this?

The sample dashboard looks like this:

 

 

 

Need Help
  • bhanu_gautam's avatar
    bhanu_gautam
    2 years ago

    Anonymous , You can try this in Power query

     

    Select the table you want to work with.
    Select the columns Appeared, Disappeared, and Component.
    Go to the "Add Column" tab and click on "Merge Columns".
    Choose a separator (e.g., a comma) and provide a name for the new column (e.g., Key).
    Remove Duplicates:

    With the new Key column selected, go to the "Home" tab.
    Click on "Remove Rows" and then select "Remove Duplicates".
    Create a Function to Automate the Process:

    Go to the "Home" tab and click on "Advanced Editor".
    Write a function that will combine the columns and remove duplicates. Here is an example of what the function might look like:

     

    let
    Source = (inputTable as table) as table =>
    let
    // Combine columns to create a key
    AddKeyColumn = Table.AddColumn(inputTable, "Key", each Text.Combine({Text.From([Appeared]), Text.From([Disappeared]), Text.From([Component])}, ",")),

    // Remove duplicates based on the key
    RemoveDuplicates = Table.Distinct(AddKeyColumn, {"Key"})

    Apply the Function:

    Apply this function to your data source. You can do this by invoking the function on your table.

2 Replies

  • bhanu_gautam's avatar
    bhanu_gautam
    Super User
    2 years ago

    Anonymous , You can try this in Power query

     

    Select the table you want to work with.
    Select the columns Appeared, Disappeared, and Component.
    Go to the "Add Column" tab and click on "Merge Columns".
    Choose a separator (e.g., a comma) and provide a name for the new column (e.g., Key).
    Remove Duplicates:

    With the new Key column selected, go to the "Home" tab.
    Click on "Remove Rows" and then select "Remove Duplicates".
    Create a Function to Automate the Process:

    Go to the "Home" tab and click on "Advanced Editor".
    Write a function that will combine the columns and remove duplicates. Here is an example of what the function might look like:

     

    let
    Source = (inputTable as table) as table =>
    let
    // Combine columns to create a key
    AddKeyColumn = Table.AddColumn(inputTable, "Key", each Text.Combine({Text.From([Appeared]), Text.From([Disappeared]), Text.From([Component])}, ",")),

    // Remove duplicates based on the key
    RemoveDuplicates = Table.Distinct(AddKeyColumn, {"Key"})

    Apply the Function:

    Apply this function to your data source. You can do this by invoking the function on your table.