Forum Discussion
Running count starting over when the counted value is not present previous date
- 5 years ago
Hi Anonymous ,
you have to modify the code like so:
let fnIdx = (myTable as table) => let partition = Table.Buffer(myTable), dates = AllDatesWthIndex, #"Merged Queries" = Table.NestedJoin(partition, {"Date"}, dates, {"Date"}, "Changed Type", JoinKind.LeftOuter), #"Expanded Changed Type" = Table.ExpandTableColumn(#"Merged Queries", "Changed Type", {"Index"}, {"Index"}), Custom1 = Table.ToColumns(#"Expanded Changed Type"), IndexedTable = Table.Buffer( Table.Sort(Table.FromColumns(Custom1 & { {null} & List.RemoveLastN(List.Last(Custom1),1) }, {"ID", "Date", "Warehouse", "Index", "PrevIndex"}), "Index") ), Initial = IndexedTable{0} & [Counter = 0, Idx = 1], Custom3 = List.Generate( ()=> Initial, each [Counter] < Table.RowCount(IndexedTable), each [ Counter = [Counter] + 1, Idx = if IndexedTable[Index]{Counter} - IndexedTable[PrevIndex]{Counter} = 1 then [Idx] + 1 else 1 ]), #"Converted to Table" = Table.FromList(Custom3, Splitter.SplitByNothing(), null, null, ExtraValues.Error), #"Expanded Column1" = Table.ExpandRecordColumn(#"Converted to Table", "Column1", {"Idx"}, {"Idx"})[Idx], Custom2 = Table.FromColumns( Table.ToColumns( IndexedTable ) & { #"Expanded Column1" }) in Custom2, Origine = dataSample, Selection = Table.FirstN(Origine,CountRecordsImke), AllDates=Table.Buffer( Table.Distinct(Table.SelectColumns(Selection, {"Date"})) ), AllDatesWthIndex = Table.AddIndexColumn(AllDates, "Index", 0, 1, Int64.Type), AllProducts = Origine, #"Grouped Rows" = Table.Group(Selection, {"ProductID"}, {{"ProductPartition", each _}}), #"Added Custom" = Table.AddColumn(#"Grouped Rows", "Custom", each fnIdx([ProductPartition])), #"Expanded Custom" = Table.ExpandTableColumn(#"Added Custom", "Custom", {"Column2", "Column3", "Column6"}, {"Date", "Warehouse", "Index"}), #"Removed Columns" = Table.RemoveColumns(#"Expanded Custom",{"ProductPartition"}), #"Changed Type" = Table.TransformColumnTypes(#"Removed Columns",{{"Date", type date}, {"Index", Int64.Type}, {"ProductID", type text}}) in #"Changed Type"Please also see the file attached.
- Anonymous5 years ago
Hi Anonymous .
First of all, I congratulate you for being an excellent tester. But then I tell you that I do not agree on the expected result that you indicate on the demo_real file. I am attaching a screen with the result that, from your previous descriptions, I would expect. I have corrected the script and am always curious to know how long it takes to work your complete file.
The script assumes that the data are sorted by date, otherwise you have to add a statement to that effect at the beginning.
let
Origine = Table.FromRows(Json.Document(Binary.Decompress(Binary.FromText("i45WclTSUTLTM9QzMjAyUIrViVZyQhdwRhcAabFE12KJrgVDhaEhuiGGJhhqTNCNQRUB6zLFUIMQiQUA", BinaryEncoding.Base64), Compression.Deflate)), let _t = ((type nullable text) meta [Serialized.Text = true]) in type table [prid = _t, date = _t]),
#"Raggruppate righe1" = Table.Group(Origine, {"date"}, {{"all", each _[prid]}}),
#"Rimosse altre colonne" = Table.SelectColumns(#"Raggruppate righe1",{"all"}),
all = #"Rimosse altre colonne"[all],
lg=List.Generate(
()=>[A=0,B=0,C=0,n=0],
each [n]<=List.Count(all),
each [A=if List.Contains(all{[n]},"A")then [A]+1 else 0,
B=if List.Contains(all{[n]},"B")then [B]+1 else 0,
C=if List.Contains(all{[n]},"C")then [C]+1 else 0,
n=[n]+1],
each Table.Transpose(Table.DemoteHeaders(Table.FromRecords({_[[A],[B],[C]]}))
)),
tc=Table.Combine(lg),
#"Filtrate righe" = Table.SelectRows(tc, each ([Column2] <> 0))
in
#"Filtrate righe"
Try using a pattern like this. it's tangled (and for this reason I suggest you toi test carefully the script), but I can't find anything simpler and more direct for the moment.
- Anonymous5 years agoNot applicable
Hi,
thank you very much for an advice! Could you please add a little bit of information how to use the code, so the extra runing count column is created? Unfortunately I don't know how to use it properly so it doesn't work at the moment. I'm trying to find out the way myself using various tutorials but without success.
Also, the real dataset contains 10K+ of rows with hundreds of distinct product IDs which are all 14-digit numbers. On top of that, these IDs are added new every month so I need the solution to be flexible whenever there's a change. Do you think I'd be able to use your code if implemented correctly? I'm sorry I didn't mention that earlier but since I don't have much experience I supposed it was not that important but now I guess it probably is. Once again thanks a lot.
Honzik
- Anonymous5 years agoNot applicable
try these versions, should perfom better (and the second better than first)
let Origine = Table.FromRows(Json.Document(Binary.Decompress(Binary.FromText("i45WclTSUTLTM9QzMjAyUIrViVZyQhdwRhcAabFE12KJrgVDhaEhuiGGJhhqTNCNQRVxwRABm2OKoQshEgsA", BinaryEncoding.Base64), Compression.Deflate)), let _t = ((type nullable text) meta [Serialized.Text = true]) in type table [prid = _t, date = _t]), d=Origine[date], all = List.Buffer(Table.Group(Origine, {"date"}, {{"all", each _[prid]}})[all]), nall=List.Count(all), allprid=List.Buffer(List.Union(all)), nprid=List.Count(allprid), rec0=Record.FromList(List.Repeat({0},nprid),allprid), lg=List.Generate( ()=>rec0&[n=0], each [n]<=nall, each Record.TransformFields(_, List.Transform(allprid, (p)=> {p, (x)=>if List.Contains(all{[n]},p) then x+1 else 0}))&[n=[n]+1], each Record.ToTable(Record.RemoveFields(_,"n")) ), tc=Table.Combine(lg), #"Filtrate righe" = Table.SelectRows(tc, each ([Value] <> 0)), ttc=Table.FromColumns({d}&Table.ToColumns(#"Filtrate righe"),{"date","grpid","idx"}) in ttclet Origine = Table.FromRows(Json.Document(Binary.Decompress(Binary.FromText("i45WclTSUTLTM9QzMjAyUIrViVZyQhdwRhcAabFE12KJrgVDhaEhuiGGJhhqTNCNQRVxwRABm2OKoQshEgsA", BinaryEncoding.Base64), Compression.Deflate)), let _t = ((type nullable text) meta [Serialized.Text = true]) in type table [prid = _t, date = _t]), d=Origine[date], all = List.Buffer(Table.Group(Origine, {"date"}, {{"all", each _[prid]}})[all]), nall=List.Count(all), allprid=List.Buffer(List.Union(all)), nprid=List.Count(allprid), rec0=Record.FromList(List.Repeat({0},nprid),allprid), lg=List.Generate( ()=>rec0&[n=0], each [n]<=nall, each Record.TransformFields(_, List.Transform(allprid, (p)=> {p, (x)=>if List.Contains(all{[n]},p) then x+1 else 0}))&[n=[n]+1] ), lglb=List.Buffer(lg), tc=Table.Combine(List.Transform(lglb, each Record.ToTable(_))), #"Filtrate righe" = Table.SelectRows(tc, each ([Value] <> 0 and [Name]<>"n")), ttc=Table.FromColumns({d}&Table.ToColumns(#"Filtrate righe"),{"date","grpid","idx"}) in ttcPS
I came up with a different idea on how to do it, but I need time to implement it. In this case without using the list.generate function which works one record at a time and therefore could be much faster.
PPS
If you can upload a table with your productIDs and dates, I can do some tests. I don't think publishing productIDs and a list of dates is a problem (since they are just numeric codes), is it? eventually you can also "encode" them and dates by removing or adding something to all.
- Anonymous5 years agoNot applicable
Hi,
this is the link where I uploaded the dataset - https://drive.google.com/file/d/16ooEjU8S-T2dPVic9cf3QvntIpUGl41a/view?usp=sharing There's one more column showing Warehouse ID I was asked to add but there's no need to calculate with that value - it's just supposed to be displayed in the final table. I'm slowly starting to understand how some of the M code works but still struggling to do this ony my own.
I have not tried the last script you posted but the one before worked though it took hours for my laptop to process more data. Once again big thanks for everything, you've already helped a lot.Honzik
- Anonymous5 years agoNot applicable
....
this adds the code to populate the table with the date column
let Origine = Table.FromRows(Json.Document(Binary.Decompress(Binary.FromText("i45WclTSUTLTM9QzMjAyUIrViVZyQhdwRhcAabFE12KJrgVDhaEhuiGGJhhqTNCNQRUB6zLFUIMQiQUA", BinaryEncoding.Base64), Compression.Deflate)), let _t = ((type nullable text) meta [Serialized.Text = true]) in type table [prid = _t, date = _t]), d=Origine[date], #"Raggruppate righe1" = Table.Group(Origine, {"date"}, {{"all", each _[prid]}}), #"Rimosse altre colonne" = Table.SelectColumns(#"Raggruppate righe1",{"all"}), all = #"Rimosse altre colonne"[all], lg=List.Generate( ()=>[A=0,B=0,C=0,n=0], each [n]<=List.Count(all), each [A=if List.Contains(all{[n]},"A")then [A]+1 else 0, B=if List.Contains(all{[n]},"B")then [B]+1 else 0, C=if List.Contains(all{[n]},"C")then [C]+1 else 0, n=[n]+1], each Table.Transpose(Table.DemoteHeaders(Table.FromRecords({_[[A],[B],[C]]})) )), tc=Table.Combine(lg), #"Filtrate righe" = Table.SelectRows(tc, each ([Column2] <> 0)), ttc=Table.FromColumns({d}&Table.ToColumns(#"Filtrate righe"),{"date","grpid","idx"}) in ttcas for the general problem, you should provide some more detailed examples (with fake data).
you should explain better in the more general cases the rule of formation of groups.
do you confirm, for example, that the productid groups to be considered are identified as those associated with each date?
Finally, if you explained more generally what you start from and what you want to achieve, perhaps we could find a different way from that of having to index the products in this complicated way.