Forum Discussion

sauliux's avatar
sauliux
Frequent Visitor
3 years ago
Solved

Grouping Times Series Data

In short I have manually recorded luxmeter data with values of interest, and 0 values inbetween (in order to keep track of which measurements are for which sample).
I want to group this time series data highs in a serial fashion, and I am struggling to figure out how to write the conditional statements for the "Custom Column" (I suspect the magical "List.Generate" and "Custom Function" have something to do with it).
So far I was able to "find" the data boundaries of interest ("FALSE" values in the "Data Boundries" column from the image below), but serial numbering the time series data highs is turning out to be a challenge:

I was hoping that declaring a variable (in this case "n"), and then in an if conditional statement adding one to it (n=n+1) would allow me to number my groups. Apparently the variable is immutable?
For the current stage I am hoping to achieve the following:

At a later stage I would filter out the 0 values from the "Custom" column, group the data by "Data Boundries" column, average the the 5 "middle" values  in the grouped "Data Boundries" tables (last and first 2-3 measurements of each group contain discrepancies)
Suggestions how incorporate n=n+1 into the custom column function would be highly appreciated. 

  • Hi sauliux ,
    you can do this with special grouping parameters: (If you are interested: Table.Group: Exploring the 5th element in Power BI and Power Query – The BIccountant )

     

    let
      Source = = Excel.CurrentWorkbook(){[Name="Table1"]}[Content], 
      #"Changed Type" = Table.TransformColumnTypes(
        Source, 
        {
          {"Record Time", type text}, 
          {"Solar", Int64.Type}, 
          {"Index", Int64.Type}, 
          {"Custom", Int64.Type}, 
          {"Custom1", Int64.Type}, 
          {"Data_Boundries", type any}
        }
      ), 
      #"Grouped Rows" = Table.Group(
        #"Changed Type", 
        {"Data_Boundries"}, 
        {{"Count", each _}}, 
        GroupKind.Local, 
        (group, current) => Number.From(current[Data_Boundries] = false)
      ), 
      #"Added Index" = Table.AddIndexColumn(#"Grouped Rows", "Index", 1, 1, Int64.Type), 
      #"Expanded Count" = Table.ExpandTableColumn(
        #"Added Index", 
        "Count", 
        {"Record Time", "Solar", "Index", "Custom", "Custom1", "Data_Boundries"}, 
        {"Record Time", "Solar", "Index.1", "Custom", "Custom1", "Data_Boundries.1"}
      ), 
      #"Added Custom" = Table.AddColumn(
        #"Expanded Count", 
        "Custom.1", 
        each if [Data_Boundries.1] = 0 then 0 else [Index]
      ), 
      #"Removed Columns" = Table.RemoveColumns(
        #"Added Custom", 
        {"Data_Boundries", "Data_Boundries.1", "Index"}
      ), 
      #"Renamed Columns" = Table.RenameColumns(#"Removed Columns", {{"Custom.1", "Data_Boundries"}})
    in
      #"Renamed Columns"

     


    See also attached file.

     

7 Replies

  • ImkeF's avatar
    ImkeF
    Community Champion

    Hi sauliux ,
    you can do this with special grouping parameters: (If you are interested: Table.Group: Exploring the 5th element in Power BI and Power Query – The BIccountant )

     

    let
      Source = = Excel.CurrentWorkbook(){[Name="Table1"]}[Content], 
      #"Changed Type" = Table.TransformColumnTypes(
        Source, 
        {
          {"Record Time", type text}, 
          {"Solar", Int64.Type}, 
          {"Index", Int64.Type}, 
          {"Custom", Int64.Type}, 
          {"Custom1", Int64.Type}, 
          {"Data_Boundries", type any}
        }
      ), 
      #"Grouped Rows" = Table.Group(
        #"Changed Type", 
        {"Data_Boundries"}, 
        {{"Count", each _}}, 
        GroupKind.Local, 
        (group, current) => Number.From(current[Data_Boundries] = false)
      ), 
      #"Added Index" = Table.AddIndexColumn(#"Grouped Rows", "Index", 1, 1, Int64.Type), 
      #"Expanded Count" = Table.ExpandTableColumn(
        #"Added Index", 
        "Count", 
        {"Record Time", "Solar", "Index", "Custom", "Custom1", "Data_Boundries"}, 
        {"Record Time", "Solar", "Index.1", "Custom", "Custom1", "Data_Boundries.1"}
      ), 
      #"Added Custom" = Table.AddColumn(
        #"Expanded Count", 
        "Custom.1", 
        each if [Data_Boundries.1] = 0 then 0 else [Index]
      ), 
      #"Removed Columns" = Table.RemoveColumns(
        #"Added Custom", 
        {"Data_Boundries", "Data_Boundries.1", "Index"}
      ), 
      #"Renamed Columns" = Table.RenameColumns(#"Removed Columns", {{"Custom.1", "Data_Boundries"}})
    in
      #"Renamed Columns"

     


    See also attached file.

     

    • sauliux's avatar
      sauliux
      Frequent Visitor

      ImkeF That was so smooth. Thank you very much.
      I suspect that a lot of the M code List.Accumulate() and/or List.Generate() loop examples that I have come across could have been done using your Table.Group() technique.

      • sauliux's avatar
        sauliux
        Frequent Visitor

        "after" sample data

        Record TimeSolarIndexCustomCustom.1Data Boundries
        18-10-2022 14:01:012791111
        18-10-2022 14:01:02402,82101
        18-10-2022 14:01:0303000
        18-10-2022 14:01:0404000
        18-10-2022 14:01:0505000
        18-10-2022 14:01:0606000
        18-10-2022 14:01:0707000
        18-10-2022 14:01:0808000
        18-10-2022 14:01:0909000
        18-10-2022 14:01:10010000
        18-10-2022 14:01:11011012
        18-10-2022 14:01:12513,712112
        18-10-2022 14:01:13473,713112
        18-10-2022 14:01:14473,314112
        18-10-2022 14:01:15482,215112
        18-10-2022 14:01:16480,516112
        18-10-2022 14:01:17480,517112
        18-10-2022 14:01:18481,618112
        18-10-2022 14:01:19480,519112
        18-10-2022 14:01:20478,820112
        18-10-2022 14:01:21485,721102
        18-10-2022 14:01:22022000
        18-10-2022 14:01:23023000
        18-10-2022 14:01:24024000
        18-10-2022 14:01:25025000
        18-10-2022 14:01:26026000
        18-10-2022 14:01:27027000
        18-10-2022 14:01:28028000
        18-10-2022 14:01:29029000
        18-10-2022 14:01:30030000
        18-10-2022 14:01:31031013
        18-10-2022 14:01:32291,832113
        18-10-2022 14:01:3332133113
        18-10-2022 14:01:34321,634113
        18-10-2022 14:01:35321,635113
        18-10-2022 14:01:36321,336113
        18-10-2022 14:01:37321,337113
        18-10-2022 14:01:38321,338113
        18-10-2022 14:01:3932139113
        18-10-2022 14:01:40321,340113
        18-10-2022 14:01:41321,341113
        18-10-2022 14:01:42321,642103
        18-10-2022 14:01:43043000
        18-10-2022 14:01:44044000
        18-10-2022 14:01:45045000
        18-10-2022 14:01:46046000
        18-10-2022 14:01:47047000