Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

The Power BI DataViz World Championships are on! With four chances to enter, you could win a spot in the LIVE Grand Finale in Las Vegas. Show off your skills.

Reply
analyst123
Frequent Visitor

Generating a filter to remove duplicates

Hello,

In my data source, there are three tables: Appeared, Disappeared and Component. I need to remove the duplicates if all of these three tables are the same for different rows. But I also keep adding data everyday, so I need a function/code that will automatically remove duplicates as and when I add more data.

I have combined the three columns into one and called it key. But even if I just try to remove duplicates from the key (transform data), it does not work. Can someone please help me with this?

The sample dashboard looks like this:

analyst123_0-1724941688693.png

 

 

 

1 ACCEPTED SOLUTION
bhanu_gautam
Super User
Super User

@analyst123 , You can try this in Power query

 

Select the table you want to work with.
Select the columns Appeared, Disappeared, and Component.
Go to the "Add Column" tab and click on "Merge Columns".
Choose a separator (e.g., a comma) and provide a name for the new column (e.g., Key).
Remove Duplicates:

With the new Key column selected, go to the "Home" tab.
Click on "Remove Rows" and then select "Remove Duplicates".
Create a Function to Automate the Process:

Go to the "Home" tab and click on "Advanced Editor".
Write a function that will combine the columns and remove duplicates. Here is an example of what the function might look like:

 

let
Source = (inputTable as table) as table =>
let
// Combine columns to create a key
AddKeyColumn = Table.AddColumn(inputTable, "Key", each Text.Combine({Text.From([Appeared]), Text.From([Disappeared]), Text.From([Component])}, ",")),

// Remove duplicates based on the key
RemoveDuplicates = Table.Distinct(AddKeyColumn, {"Key"})

Apply the Function:

Apply this function to your data source. You can do this by invoking the function on your table.




Did I answer your question? Mark my post as a solution! And Kudos are appreciated

Proud to be a Super User!




LinkedIn






View solution in original post

2 REPLIES 2
_AAndrade
Super User
Super User

Hi @analyst123,

Did you try to do that in power query?





Did I answer your question? Mark my post as a solution! Kudos are welcome.

Proud to be a Super User!




bhanu_gautam
Super User
Super User

@analyst123 , You can try this in Power query

 

Select the table you want to work with.
Select the columns Appeared, Disappeared, and Component.
Go to the "Add Column" tab and click on "Merge Columns".
Choose a separator (e.g., a comma) and provide a name for the new column (e.g., Key).
Remove Duplicates:

With the new Key column selected, go to the "Home" tab.
Click on "Remove Rows" and then select "Remove Duplicates".
Create a Function to Automate the Process:

Go to the "Home" tab and click on "Advanced Editor".
Write a function that will combine the columns and remove duplicates. Here is an example of what the function might look like:

 

let
Source = (inputTable as table) as table =>
let
// Combine columns to create a key
AddKeyColumn = Table.AddColumn(inputTable, "Key", each Text.Combine({Text.From([Appeared]), Text.From([Disappeared]), Text.From([Component])}, ",")),

// Remove duplicates based on the key
RemoveDuplicates = Table.Distinct(AddKeyColumn, {"Key"})

Apply the Function:

Apply this function to your data source. You can do this by invoking the function on your table.




Did I answer your question? Mark my post as a solution! And Kudos are appreciated

Proud to be a Super User!




LinkedIn






Helpful resources

Announcements
Las Vegas 2025

Join us at the Microsoft Fabric Community Conference

March 31 - April 2, 2025, in Las Vegas, Nevada. Use code MSCUST for a $150 discount!

FebPBI_Carousel

Power BI Monthly Update - February 2025

Check out the February 2025 Power BI update to learn about new features.

Feb2025 Sticker Challenge

Join our Community Sticker Challenge 2025

If you love stickers, then you will definitely want to check out our Community Sticker Challenge!

Feb2025 NL Carousel

Fabric Community Update - February 2025

Find out what's new and trending in the Fabric community.