Forum Discussion
Impactful IndividualData overflow error when the "List.Accumulate" function was used to process data over 2000 rows
Curt Hagenlocher's argument seems interesting to me.
<<...
But broadly speaking, M is a lazy language and some of the things you compute actually result in promises instead of immediate values. If the combination function passed to List.Accumulate results in a promise, then the result of accumulating over your list is a promise on top of a promise on top of... etc. At some point, when the actual value is calculated, it's done recursively -- the chain of promises is too long and the stack flows over.
>>
it is not clear to me where and why the problem of overflow with the list.accumulate function is needed (apart from the one in which an infinite loop is created).
But for the original problem, a faster solution than that using list.accumulate might be to turn the initial table into a record ( i called sorgente) where the SN column are the fields and the Value column are the values.
after with the following expression get the final table
S = Table.FromRows(List.Transform(List.Transform({1..200000},each Text.From(_)),each {_,Record.FieldOrDefault(sorgente,_,"0")}),{"SN","Value"})
it seems the problem you ran into is already known
- ziying35
Anonymous
I think it is the problem at the bottom of the program. When the amount of data is less than 2000, the speed is still very fast. When the amount of data is more than 2000 rows, the speed will slow down and even make mistakes, I read the post linked above, but Microsoft hasn't fixed the problem. Thanks
I made a little experiment with those two basic expressions:
sum = List.Accumulate(Sorgn,0,(s,c)=> s+c), combine = List.Accumulate(Sorgn,{},(s,c)=> List.Combine({s,{c}}))the first one it can even calculate millions of terms, instead of the second overflowing after about 64000/64500 elements
this
List.Accumulate(Sorgn,[a={0}],(s,c)=> s&[a=s[a]&{c}])and this
List.Accumulate(Sorgn,[a=0],(s,c)=> s&[a=s[a]+c])goes in overflow just after 7000 elements.
- ziying35
Iterating over a record with the List.Accumulate function usually doesn't go wrong, but it does when it comes to combining lists or tables etc.
In our PQ group, we usually only dare to use it to deal with some data under 2000 lines
