Forum Discussion
Running count starting over when the counted value is not present previous date
- 5 years ago
Hi Anonymous ,
you have to modify the code like so:
let fnIdx = (myTable as table) => let partition = Table.Buffer(myTable), dates = AllDatesWthIndex, #"Merged Queries" = Table.NestedJoin(partition, {"Date"}, dates, {"Date"}, "Changed Type", JoinKind.LeftOuter), #"Expanded Changed Type" = Table.ExpandTableColumn(#"Merged Queries", "Changed Type", {"Index"}, {"Index"}), Custom1 = Table.ToColumns(#"Expanded Changed Type"), IndexedTable = Table.Buffer( Table.Sort(Table.FromColumns(Custom1 & { {null} & List.RemoveLastN(List.Last(Custom1),1) }, {"ID", "Date", "Warehouse", "Index", "PrevIndex"}), "Index") ), Initial = IndexedTable{0} & [Counter = 0, Idx = 1], Custom3 = List.Generate( ()=> Initial, each [Counter] < Table.RowCount(IndexedTable), each [ Counter = [Counter] + 1, Idx = if IndexedTable[Index]{Counter} - IndexedTable[PrevIndex]{Counter} = 1 then [Idx] + 1 else 1 ]), #"Converted to Table" = Table.FromList(Custom3, Splitter.SplitByNothing(), null, null, ExtraValues.Error), #"Expanded Column1" = Table.ExpandRecordColumn(#"Converted to Table", "Column1", {"Idx"}, {"Idx"})[Idx], Custom2 = Table.FromColumns( Table.ToColumns( IndexedTable ) & { #"Expanded Column1" }) in Custom2, Origine = dataSample, Selection = Table.FirstN(Origine,CountRecordsImke), AllDates=Table.Buffer( Table.Distinct(Table.SelectColumns(Selection, {"Date"})) ), AllDatesWthIndex = Table.AddIndexColumn(AllDates, "Index", 0, 1, Int64.Type), AllProducts = Origine, #"Grouped Rows" = Table.Group(Selection, {"ProductID"}, {{"ProductPartition", each _}}), #"Added Custom" = Table.AddColumn(#"Grouped Rows", "Custom", each fnIdx([ProductPartition])), #"Expanded Custom" = Table.ExpandTableColumn(#"Added Custom", "Custom", {"Column2", "Column3", "Column6"}, {"Date", "Warehouse", "Index"}), #"Removed Columns" = Table.RemoveColumns(#"Expanded Custom",{"ProductPartition"}), #"Changed Type" = Table.TransformColumnTypes(#"Removed Columns",{{"Date", type date}, {"Index", Int64.Type}, {"ProductID", type text}}) in #"Changed Type"Please also see the file attached.
- Anonymous5 years ago
Hi Anonymous .
First of all, I congratulate you for being an excellent tester. But then I tell you that I do not agree on the expected result that you indicate on the demo_real file. I am attaching a screen with the result that, from your previous descriptions, I would expect. I have corrected the script and am always curious to know how long it takes to work your complete file.
The script assumes that the data are sorted by date, otherwise you have to add a statement to that effect at the beginning.
Hi Anonymous and Anonymous ,
you might be interested in an approach that runs the full dataset in seconds instead of minutes:
let
fnIdx =
(myTable as table) =>
let
partition = Table.Buffer(myTable),
dates = AllDatesWthIndex,
#"Merged Queries" = Table.NestedJoin(partition, {"Date"}, dates, {"Date"}, "Changed Type", JoinKind.LeftOuter),
#"Expanded Changed Type" = Table.ExpandTableColumn(#"Merged Queries", "Changed Type", {"Index"}, {"Index"}),
Custom1 = Table.ToColumns(#"Expanded Changed Type"),
IndexedTable = Table.Buffer( Table.FromColumns(Custom1 & { {null} & List.RemoveLastN(List.Last(Custom1),1) }, {"ID", "Date", "Index", "PrevIndex"}) ),
Initial = IndexedTable{0} & [Counter = 0, Idx = 1],
Custom3 = List.Generate( ()=> Initial,
each [Counter] < Table.RowCount(IndexedTable),
each [ Counter = [Counter] + 1,
Idx = if IndexedTable[Index]{Counter} - IndexedTable[PrevIndex]{Counter} = 1 then [Idx] + 1 else 1
]),
#"Converted to Table" = Table.FromList(Custom3, Splitter.SplitByNothing(), null, null, ExtraValues.Error),
#"Expanded Column1" = Table.ExpandRecordColumn(#"Converted to Table", "Column1", {"Idx"}, {"Idx"})[Idx],
Custom2 = Table.FromColumns( Table.ToColumns( IndexedTable ) & { #"Expanded Column1" })
in
Custom2,
Origine = dataSample,
Selection = Table.FirstN(Origine,CountRecordsImke),
AllDates=Table.Buffer( Table.Distinct(Table.SelectColumns(Selection, {"Date"})) ),
AllDatesWthIndex = Table.AddIndexColumn(AllDates, "Index", 0, 1, Int64.Type),
AllProducts = Origine,
#"Grouped Rows" = Table.Group(Selection, {"ProductID"}, {{"ProductPartition", each _}}),
#"Added Custom" = Table.AddColumn(#"Grouped Rows", "Custom", each fnIdx([ProductPartition])),
#"Expanded Custom" = Table.ExpandTableColumn(#"Added Custom", "Custom", {"Column2", "Column5"}, {"Date", "Index"}),
#"Removed Columns" = Table.RemoveColumns(#"Expanded Custom",{"ProductPartition"}),
#"Changed Type" = Table.TransformColumnTypes(#"Removed Columns",{{"Date", type date}, {"Index", Int64.Type}, {"ProductID", type text}})
in
#"Changed Type"
It runs the logic on the level of the individual ProductIDs like I've described here: Performance Tip: Partition your tables at crossjoins where possible - PowerQuery PowerBI – The BIccountant
and uses a shifted index method which also makes comparisons very fast: Fast and easy way to reference previous or next rows in Power Query or Power BI – The BIccountant
Power Query doesn't have to be slow if you do it the right way. I've collected some tipps and tricks to improve performance here: Speed/Performance aspects – The BIccountant
Hi ImkeF,
thank you very much for your solution. I'll go through it so I can better undestand the concept but looks like great approach to it! One more question - could you please give me a hint how to run it using all columns (including Warehouse) so the warehouse values remain there when it's finished? (Dateset including that column is here https://drive.google.com/file/d/16ooEjU8S-T2dPVic9cf3QvntIpUGl41a/view?usp=sharing.
Thanks a lot!
Honzik