Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

Score big with last-minute savings on the final tickets to FabCon Vienna. Secure your discount

Reply
MatusiSK
New Member

Remove duplicate entry based on calculation between matching rows (datetime difference)

Hey Team!

 

I have very complex problem I'm struggling to resolve. I assume there is several ways to do it. I'm looking for performance wise aproach that can handle milions of rows.

Goal: summarize real edit time (from Agent perspective)

Problem: When editing a Case system create duplicate record if agent goes from Case to WO. New row/record is created for same Case.

Duplicate: duplicate record is row with Work Order ID. At least part of the time edit is simultaneous with Case edit. When Agent end edit mode (End Time) then Case Edit is automatically ended usually 7-9 seconds later. The Work Order edit is thus duplicate and for tracking purposes we need only Case row

Solution: I tried chatgpt proposal adding new custom column that returns end time of previous Case edit (based on matching Case Number and Start by which is same agent. After I sorted data and used Buffer.Table to keep correct sorting.

 

#"Added End Time From Case Row" = Table.AddColumn(#"Added Index", "End Time (From Case)", each
if [Work Order] <> null then
let
CurrentCase = [#"Case Number (Case) (Case)"],
CurrentStartBy = [Start By],
// Find the first row where "Work Order" is null and Case Number and Start By match
MatchedRow = Table.SelectRows(#"Added Index", each
[#"Case Number (Case) (Case)"] = CurrentCase and
[Start By] = CurrentStartBy and
[Work Order] = null
){0}?, // Use "{0}?" to safely return the first matching row or null if no match
EndTimeValue = if MatchedRow <> null then MatchedRow[End Time] else null
in
EndTimeValue
else
null

 

Do you think there is better solution to this problem that is more performance friendly or in general better approach that consider row calculation based on same condition ? Thank you for your help!

MatusiSK_0-1729604229873.png

 

2 REPLIES 2
Anonymous
Not applicable

Hi @MatusiSK ,

The M code you provided is not complete, so I may not know whether there is room for improvement. However, for Power Query to optimize query performance, the most effective method may still be to use Group by Function.
Table.Group - PowerQuery M | Microsoft Learn

Please check whether your data can be grouped first and then perform other data transformation operations.

Best Regards,
Dino Tao
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

MatusiSK
New Member

Case NumberWork OrderCreation ModeStart ByEnd ByStart TimeEnd TimeSec. Difference
2020098993 ManualAgent 4Agent 49/2/2024 09:49:209/2/2024 10:04:11 
2020098993 AutoAgent 4SYSTEM9/2/2024 15:15:029/2/2024 15:16:15 
20200989934014643198ManualAgent 4Agent 49/2/2024 15:15:219/2/2024 15:16:087.00
20200989934014643198ManualAgent 4Agent 49/2/2024 15:16:279/2/2024 15:18:49 
2020098993 ManualAgent 4Agent 49/4/2024 08:48:029/4/2024 08:48:17 
2020098993 ManualAgent 4Agent 49/6/2024 09:17:309/6/2024 09:22:37 
2020098993 ManualAgent 4Agent 49/9/2024 09:50:509/9/2024 09:51:36 
2020098993 ManualAgent 4Agent 49/11/2024 16:28:539/11/2024 16:28:57 
2020099685 AutoAgent 5SYSTEM9/9/2024 11:26:299/9/2024 11:28:46 
20200996854014736440ManualAgent 5Agent 59/9/2024 11:26:359/9/2024 11:28:397.00
20200996854014736440ManualAgent 5Agent 59/9/2024 11:29:079/9/2024 11:30:39 
2020100256 AutoAgent 5SYSTEM9/13/2024 09:21:599/13/2024 09:23:13 
20201002564014816836ManualAgent 5Agent 59/13/2024 09:22:189/13/2024 09:23:067.00
20201002564014816836ManualAgent 5Agent 59/13/2024 09:23:249/13/2024 09:24:52 
2020102423 ManualAgent 6Agent 69/12/2024 10:50:599/12/2024 10:52:21 
2020102423 ManualAgent 6Agent 69/11/2024 13:37:359/11/2024 13:38:22 
2020116155 AutoAgent 7SYSTEM9/2/2024 11:17:369/2/2024 11:32:39 
2020116155 AutoAgent 7SYSTEM9/2/2024 12:59:049/2/2024 12:59:43 
20201161554014640968ManualAgent 7Agent 79/2/2024 12:59:199/2/2024 12:59:367.00
20201161554014640968ManualAgent 7Agent 79/2/2024 13:00:019/2/2024 13:01:01 
2020116155 AutoAgent 5SYSTEM9/11/2024 09:18:479/11/2024 09:25:20 
20201161554014777495ManualAgent 5Agent 59/11/2024 09:24:209/11/2024 09:25:119.00
20201161554014777495ManualAgent 5Agent 59/11/2024 09:25:289/11/2024 09:26:54 
2020123991 ManualAgent 8Agent 89/2/2024 15:29:049/2/2024 15:33:36 
2020123991 AutoAgent 8SYSTEM9/10/2024 15:50:109/10/2024 15:54:01 
20201239914014761470ManualAgent 8Agent 89/10/2024 15:52:549/10/2024 15:53:547.00
20201239914014761470ManualAgent 8Agent 89/10/2024 15:54:039/10/2024 15:54:17 
2020143741 AutoAgent 6SYSTEM9/9/2024 08:32:419/9/2024 08:50:33 
2020143741 ManualAgent 6Agent 69/6/2024 10:41:549/6/2024 10:43:44 
2020148485 AutoAgent 9SYSTEM9/3/2024 16:15:449/3/2024 16:30:47 
2020148485 AutoAgent 9SYSTEM9/3/2024 16:33:559/3/2024 17:01:57 
2020148485 AutoAgent 9SYSTEM9/3/2024 17:15:559/3/2024 17:30:57 
2020151763 AutoAgent 9SYSTEM9/2/2024 10:54:429/2/2024 10:55:50 
20201517634014638922ManualAgent 9Agent 99/2/2024 10:54:579/2/2024 10:55:419.00
20201517634014638922ManualAgent 9Agent 99/2/2024 10:56:179/2/2024 10:56:39 
2020152082 AutoAgent 10SYSTEM9/2/2024 12:31:439/2/2024 12:32:35 
20201520824014640527ManualAgent 10Agent 109/2/2024 12:32:049/2/2024 12:32:28 
20201520824014640527ManualAgent 10Agent 109/2/2024 12:32:409/2/2024 12:35:46 
20201520824014640527ManualAgent 10Agent 109/2/2024 12:35:589/2/2024 12:36:11 
2020155768 AutoAgent 11SYSTEM9/11/2024 12:22:009/11/2024 12:23:22 
20201557684014780830ManualAgent 11Agent 119/11/2024 12:22:539/11/2024 12:23:157.00
20201557684014780830ManualAgent 11Agent 119/11/2024 12:23:309/11/2024 12:24:03 

 

Table with sample data

Helpful resources

Announcements
August Power BI Update Carousel

Power BI Monthly Update - August 2025

Check out the August 2025 Power BI update to learn about new features.

August 2025 community update carousel

Fabric Community Update - August 2025

Find out what's new and trending in the Fabric community.

Top Solution Authors