Forum Discussion
Find difference between current and previous row - performance issue
- 3 years ago
Ok, try a different approach in Power Query. Basically it entails creating a single table with the transaction time and the next transacion time. Then leave the calculations to measures (which are simple since both transaction times are on the same row of the table.
To do this, you need to sort the table by salesman and Transaction time (in ascending order), and add an index column starting at 1
Now duplicate the table and change the index order to start at 0
Now you can merge both tables by selecting salesman and index, and keep only the new transaction row from the second table:
(You can disable load for the second table since you don't need it in the model)
If you prefer to do this whole process with a single query, here is the M code:
let Source = Table.FromRows( Json.Document( Binary.Decompress( Binary.FromText( "i45WCsnPVdJRMjLSNzDUNzIwMlIwNLIyMFCK1cEmZQqT8s1MzkhMzQFJGyOkTfBLm1kZoxiMJGVuZWKKR6cFmpNMsLgWSacJfhebkGYwVMoUWRfII7EA", BinaryEncoding.Base64 ), Compression.Deflate ) ), let _t = ((type nullable text) meta [Serialized.Text = true]) in type table [Salesman = _t, #"Transaction time" = _t] ), #"Changed Type" = Table.TransformColumnTypes( Source, {{"Salesman", type text}, {"Transaction time", type datetime}} ), #"Sorted Rows" = Table.Sort( #"Changed Type", {{"Salesman", Order.Ascending}, {"Transaction time", Order.Ascending}} ), #"Added Index" = Table.AddIndexColumn(#"Sorted Rows", "Index", 1, 1, Int64.Type), Source1 = Source, #"Changed Type1" = Table.TransformColumnTypes( Source1, {{"Salesman", type text}, {"Transaction time", type datetime}} ), #"Sorted Rows1" = Table.Sort( #"Changed Type1", {{"Salesman", Order.Ascending}, {"Transaction time", Order.Ascending}} ), #"Added Index1" = Table.AddIndexColumn(#"Sorted Rows1", "Index", 0, 1, Int64.Type), #"Merged Queries" = Table.NestedJoin( #"Added Index", {"Salesman", "Index"}, #"Added Index1", {"Salesman", "Index"}, "Next transaction", JoinKind.LeftOuter ), #"Expanded Next Transaction" = Table.ExpandTableColumn( #"Merged Queries", "Next transaction", {"Transaction time"}, {"Next transaction"} ), #"Removed Columns" = Table.RemoveColumns(#"Expanded Next Transaction", {"Index"}) in #"Removed Columns"Now the measures are much simpler:
Difference in Minutes = DATEDIFF ( MAX ( 'TD on rows'[Transaction time] ), MAX ( 'TD on rows'[Next transaction] ), MINUTE )Diff vs Next transaction time (HH:MM) = VAR _MinutesDiff = DATEDIFF ( MAX ( 'TD on rows'[Transaction time] ), MAX ( 'TD on rows'[Next transaction] ), MINUTE ) VAR _FinalHours = INT ( DIVIDE ( _MinutesDiff, 60 ) ) VAR _FinalMinutes = FORMAT ( MOD ( _MinutesDiff, 60 ), "00" ) RETURN IF ( ISBLANK ( MAX ( 'TD on rows'[Next transaction] ) ), BLANK (), _FinalHours & ":" & _FinalMinutes )New file attached
Hey, I've tried your solution for retrieving previous transaction time, and although it works it's performing poorly as well.
It took 24 seconds to load full table (just name, transaction time and previous transaction) on 15k rows test dataset. On 40k set it ran out of memory, and my real dataset is 500k and might grow to 1mln+ in future.
So I assume it means I have to step back to my original data?
Ive tried something different in the meantime. I went back to power query and I've added Transaction ID to my Transactions sorted by transaction time:
Then I've duplicated Salesman/Transaction ID into new table, grouped by salesman and ended up with a list of transaction IDs sorted by datetime, which I've then expanded back:
Up to this point everything works smoothly on 500k dataset, but now comes the bottleneck:
#"Added Custom1" = Table.AddColumn(#"Sorted Rows1", "Custom", each [Transactions List]{List.PositionOf([Transactions List],[Transaction ID])+1})
It does return proper data, which is now very easy to transform into transaction time and then, with functions you've made to time difference.
However it took about 2 minutes on 40k dataset, which is improvement, but obviously still too slow for my real data, as one of salesman has 60k transactions, meaning list of 60k length it has to search through...
PS: Oh it might get little confusing since now Ive been looking for next transaction instead of previous, but my final target is simply datetime difference, doesn't matter whether vs previous or next.
Ok, try a different approach in Power Query. Basically it entails creating a single table with the transaction time and the next transacion time. Then leave the calculations to measures (which are simple since both transaction times are on the same row of the table.
To do this, you need to sort the table by salesman and Transaction time (in ascending order), and add an index column starting at 1
Now duplicate the table and change the index order to start at 0
Now you can merge both tables by selecting salesman and index, and keep only the new transaction row from the second table:
(You can disable load for the second table since you don't need it in the model)
If you prefer to do this whole process with a single query, here is the M code:
let
Source = Table.FromRows(
Json.Document(
Binary.Decompress(
Binary.FromText(
"i45WCsnPVdJRMjLSNzDUNzIwMlIwNLIyMFCK1cEmZQqT8s1MzkhMzQFJGyOkTfBLm1kZoxiMJGVuZWKKR6cFmpNMsLgWSacJfhebkGYwVMoUWRfII7EA",
BinaryEncoding.Base64
),
Compression.Deflate
)
),
let
_t = ((type nullable text) meta [Serialized.Text = true])
in
type table [Salesman = _t, #"Transaction time" = _t]
),
#"Changed Type" = Table.TransformColumnTypes(
Source,
{{"Salesman", type text}, {"Transaction time", type datetime}}
),
#"Sorted Rows" = Table.Sort(
#"Changed Type",
{{"Salesman", Order.Ascending}, {"Transaction time", Order.Ascending}}
),
#"Added Index" = Table.AddIndexColumn(#"Sorted Rows", "Index", 1, 1, Int64.Type),
Source1 = Source,
#"Changed Type1" = Table.TransformColumnTypes(
Source1,
{{"Salesman", type text}, {"Transaction time", type datetime}}
),
#"Sorted Rows1" = Table.Sort(
#"Changed Type1",
{{"Salesman", Order.Ascending}, {"Transaction time", Order.Ascending}}
),
#"Added Index1" = Table.AddIndexColumn(#"Sorted Rows1", "Index", 0, 1, Int64.Type),
#"Merged Queries" = Table.NestedJoin(
#"Added Index",
{"Salesman", "Index"},
#"Added Index1",
{"Salesman", "Index"},
"Next transaction",
JoinKind.LeftOuter
),
#"Expanded Next Transaction" = Table.ExpandTableColumn(
#"Merged Queries",
"Next transaction",
{"Transaction time"},
{"Next transaction"}
),
#"Removed Columns" = Table.RemoveColumns(#"Expanded Next Transaction", {"Index"})
in
#"Removed Columns"
Now the measures are much simpler:
Difference in Minutes =
DATEDIFF (
MAX ( 'TD on rows'[Transaction time] ),
MAX ( 'TD on rows'[Next transaction] ),
MINUTE
)
Diff vs Next transaction time (HH:MM) =
VAR _MinutesDiff =
DATEDIFF (
MAX ( 'TD on rows'[Transaction time] ),
MAX ( 'TD on rows'[Next transaction] ),
MINUTE
)
VAR _FinalHours =
INT ( DIVIDE ( _MinutesDiff, 60 ) )
VAR _FinalMinutes =
FORMAT ( MOD ( _MinutesDiff, 60 ), "00" )
RETURN
IF (
ISBLANK ( MAX ( 'TD on rows'[Next transaction] ) ),
BLANK (),
_FinalHours & ":" & _FinalMinutes
)
New file attached
- OzzyM913 years agoFrequent Visitor
It works great now! Takes a while longer to load data but measures and visuals are calculated instantly 😎 Thank you for help and your great contribution to community!
- SebV1 year agoAdvocate I
PaulDBrown I explored your approach. Why do you merge on both Salesman and Index in step 3? Merge on Index would be sufficient als the index doesn't start over again at 1 for each new Salesman?
- PaulDBrown1 year agoCommunity Champion
If you merge only on index, you will be inputting a transaction time from salesman 2 to salesman 1 at the point where the index crosses over from one salesman to another (two different salesmen will have the same index number). By using both fields in the merge, you are ensuring the integrity of the data by salesman.
- SebV1 year agoAdvocate I
Thanks. I noticed.