Forum Discussion
Remove duplicate entry based on calculation between matching rows (datetime difference)
Hey Team!
I have very complex problem I'm struggling to resolve. I assume there is several ways to do it. I'm looking for performance wise aproach that can handle milions of rows.
Goal: summarize real edit time (from Agent perspective)
Problem: When editing a Case system create duplicate record if agent goes from Case to WO. New row/record is created for same Case.
Duplicate: duplicate record is row with Work Order ID. At least part of the time edit is simultaneous with Case edit. When Agent end edit mode (End Time) then Case Edit is automatically ended usually 7-9 seconds later. The Work Order edit is thus duplicate and for tracking purposes we need only Case row
Solution: I tried chatgpt proposal adding new custom column that returns end time of previous Case edit (based on matching Case Number and Start by which is same agent. After I sorted data and used Buffer.Table to keep correct sorting.
#"Added End Time From Case Row" = Table.AddColumn(#"Added Index", "End Time (From Case)", each
if [Work Order] <> null then
let
CurrentCase = [#"Case Number (Case) (Case)"],
CurrentStartBy = [Start By],
// Find the first row where "Work Order" is null and Case Number and Start By match
MatchedRow = Table.SelectRows(#"Added Index", each
[#"Case Number (Case) (Case)"] = CurrentCase and
[Start By] = CurrentStartBy and
[Work Order] = null
){0}?, // Use "{0}?" to safely return the first matching row or null if no match
EndTimeValue = if MatchedRow <> null then MatchedRow[End Time] else null
in
EndTimeValue
else
null
Do you think there is better solution to this problem that is more performance friendly or in general better approach that consider row calculation based on same condition ? Thank you for your help!
2 Replies
- MatusiSKNew Member
Case Number Work Order Creation Mode Start By End By Start Time End Time Sec. Difference 2020098993 Manual Agent 4 Agent 4 9/2/2024 09:49:20 9/2/2024 10:04:11 2020098993 Auto Agent 4 SYSTEM 9/2/2024 15:15:02 9/2/2024 15:16:15 2020098993 4014643198 Manual Agent 4 Agent 4 9/2/2024 15:15:21 9/2/2024 15:16:08 7.00 2020098993 4014643198 Manual Agent 4 Agent 4 9/2/2024 15:16:27 9/2/2024 15:18:49 2020098993 Manual Agent 4 Agent 4 9/4/2024 08:48:02 9/4/2024 08:48:17 2020098993 Manual Agent 4 Agent 4 9/6/2024 09:17:30 9/6/2024 09:22:37 2020098993 Manual Agent 4 Agent 4 9/9/2024 09:50:50 9/9/2024 09:51:36 2020098993 Manual Agent 4 Agent 4 9/11/2024 16:28:53 9/11/2024 16:28:57 2020099685 Auto Agent 5 SYSTEM 9/9/2024 11:26:29 9/9/2024 11:28:46 2020099685 4014736440 Manual Agent 5 Agent 5 9/9/2024 11:26:35 9/9/2024 11:28:39 7.00 2020099685 4014736440 Manual Agent 5 Agent 5 9/9/2024 11:29:07 9/9/2024 11:30:39 2020100256 Auto Agent 5 SYSTEM 9/13/2024 09:21:59 9/13/2024 09:23:13 2020100256 4014816836 Manual Agent 5 Agent 5 9/13/2024 09:22:18 9/13/2024 09:23:06 7.00 2020100256 4014816836 Manual Agent 5 Agent 5 9/13/2024 09:23:24 9/13/2024 09:24:52 2020102423 Manual Agent 6 Agent 6 9/12/2024 10:50:59 9/12/2024 10:52:21 2020102423 Manual Agent 6 Agent 6 9/11/2024 13:37:35 9/11/2024 13:38:22 2020116155 Auto Agent 7 SYSTEM 9/2/2024 11:17:36 9/2/2024 11:32:39 2020116155 Auto Agent 7 SYSTEM 9/2/2024 12:59:04 9/2/2024 12:59:43 2020116155 4014640968 Manual Agent 7 Agent 7 9/2/2024 12:59:19 9/2/2024 12:59:36 7.00 2020116155 4014640968 Manual Agent 7 Agent 7 9/2/2024 13:00:01 9/2/2024 13:01:01 2020116155 Auto Agent 5 SYSTEM 9/11/2024 09:18:47 9/11/2024 09:25:20 2020116155 4014777495 Manual Agent 5 Agent 5 9/11/2024 09:24:20 9/11/2024 09:25:11 9.00 2020116155 4014777495 Manual Agent 5 Agent 5 9/11/2024 09:25:28 9/11/2024 09:26:54 2020123991 Manual Agent 8 Agent 8 9/2/2024 15:29:04 9/2/2024 15:33:36 2020123991 Auto Agent 8 SYSTEM 9/10/2024 15:50:10 9/10/2024 15:54:01 2020123991 4014761470 Manual Agent 8 Agent 8 9/10/2024 15:52:54 9/10/2024 15:53:54 7.00 2020123991 4014761470 Manual Agent 8 Agent 8 9/10/2024 15:54:03 9/10/2024 15:54:17 2020143741 Auto Agent 6 SYSTEM 9/9/2024 08:32:41 9/9/2024 08:50:33 2020143741 Manual Agent 6 Agent 6 9/6/2024 10:41:54 9/6/2024 10:43:44 2020148485 Auto Agent 9 SYSTEM 9/3/2024 16:15:44 9/3/2024 16:30:47 2020148485 Auto Agent 9 SYSTEM 9/3/2024 16:33:55 9/3/2024 17:01:57 2020148485 Auto Agent 9 SYSTEM 9/3/2024 17:15:55 9/3/2024 17:30:57 2020151763 Auto Agent 9 SYSTEM 9/2/2024 10:54:42 9/2/2024 10:55:50 2020151763 4014638922 Manual Agent 9 Agent 9 9/2/2024 10:54:57 9/2/2024 10:55:41 9.00 2020151763 4014638922 Manual Agent 9 Agent 9 9/2/2024 10:56:17 9/2/2024 10:56:39 2020152082 Auto Agent 10 SYSTEM 9/2/2024 12:31:43 9/2/2024 12:32:35 2020152082 4014640527 Manual Agent 10 Agent 10 9/2/2024 12:32:04 9/2/2024 12:32:28 2020152082 4014640527 Manual Agent 10 Agent 10 9/2/2024 12:32:40 9/2/2024 12:35:46 2020152082 4014640527 Manual Agent 10 Agent 10 9/2/2024 12:35:58 9/2/2024 12:36:11 2020155768 Auto Agent 11 SYSTEM 9/11/2024 12:22:00 9/11/2024 12:23:22 2020155768 4014780830 Manual Agent 11 Agent 11 9/11/2024 12:22:53 9/11/2024 12:23:15 7.00 2020155768 4014780830 Manual Agent 11 Agent 11 9/11/2024 12:23:30 9/11/2024 12:24:03 Table with sample data
- AnonymousNot applicable
Hi MatusiSK ,
The M code you provided is not complete, so I may not know whether there is room for improvement. However, for Power Query to optimize query performance, the most effective method may still be to use Group by Function.
Table.Group - PowerQuery M | Microsoft Learn
Please check whether your data can be grouped first and then perform other data transformation operations.
Best Regards,
Dino Tao
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.