Forum Discussion

Ja12345's avatar
Ja12345
Regular Visitor
3 years ago
Solved

Transforming data with repeated date headers

Hello. I'm having difficulty using Power Query to transform a data report that is received in the below format.

 

I have been trying to research similar issues for the past days but haven't had success applying them.

 

The data report is always received with repeating staff ID in the first column, dates as repeating headers , and the hours worked as values. The dates go 5 columns across before wrapping underneath, always with a blank row in between each chunk of data.

 

I need to summarise the hours individual staff ID worked on certain days/for the month, so want to put this in a better format in order to analyse.

 

Sample data:

Staff ID1/02/20232/02/20233/02/20234/02/20235/02/2023
134178678
7713486786
12361267028
      
Staff ID6/02/20237/02/20238/02/20239/02/202310/02/2023
134100000
7713486868
12361288888
      
Staff ID11/02/202312/02/202313/02/202314/02/202315/02/2023
134188860
7713455568
12361200086
      
Staff ID16/02/202317/02/202318/02/202319/02/202320/02/2023
134188600
7713488600
12361288800

 

Desired output:

Staff ID134177134123612
1/02/2023786
2/02/2023867
3/02/2023670
4/02/2023782
5/02/2023868
6/02/2023088
7/02/2023068
8/02/2023088
9/02/2023068
10/02/2023088
11/02/2023850
12/02/2023850
13/02/2023850
14/02/2023668
15/02/2023086
16/02/2023888
17/02/2023888
18/02/2023668
19/02/2023000
20/02/2023000

 

Any help would be greatly appreciated!

  • First, let's see how we could handle this if we take the first four rows of the sample data. That is, start from this:

     

    From here, we can transpose the table and promote the headers to get precisely the shape you specified.

     

    Here's the code for this simplified example:

    let
        Source = Table.FromRows(Json.Document(Binary.Decompress(Binary.FromText("i45WCi5JTEtT8HRR0lEy1Dcw0jcyMDIGso2Q2MZIbBMktimcHasTrWRobGIIFDQHYgsgNoOzQZLm5kBpDBkQG6zVyNjM0AhJygDkAojmWAA=", BinaryEncoding.Base64), Compression.Deflate)), let _t = ((type nullable text) meta [Serialized.Text = true]) in type table [Column1 = _t, Column2 = _t, Column3 = _t, Column4 = _t, Column5 = _t, Column6 = _t]),
        #"Transposed Table" = Table.Transpose(Source),
        #"Promoted Headers" = Table.PromoteHeaders(#"Transposed Table", [PromoteAllScalars=true])
    in
        #"Promoted Headers"

     

    Now we just need to do the same thing for each subtable. The trick is to use Table.Split to split the table into a list of subtables, transform each table in the list just like above, and then combine the list of tables back into a single table.

     

    Here's the code for this:

    let
        Source = Table.FromRows(Json.Document(Binary.Decompress(Binary.FromText("jZGxDoMwEEN/BWVGIr5Aku5dmDtGDF34gv6/eqkqMLobGCyesCL57NbC6/Pe92F9hjFgijJJlKQsxIl4Jl4O3sYWkGboz6KqqnxwN0tR2zidf08lZQhZsSc4Hg9KVt2h8JmCFeJK/CBG9NJHIy/9+b2kr0a304O7B5cPbh9cP9z+OUA2Fywk5wI+/Zzn3gU8AHgB8ATgDcTd4Jre28C67gZ/e/sC", BinaryEncoding.Base64), Compression.Deflate)), let _t = ((type nullable text) meta [Serialized.Text = true]) in type table [Column1 = _t, Column2 = _t, Column3 = _t, Column4 = _t, Column5 = _t, Column6 = _t]),
        #"Filtered Rows" = Table.SelectRows(Source, each ([Column1] <> " ")), 
        #"Split Transpose Combine" = Table.Combine(
            List.Transform(
                Table.Split(#"Filtered Rows", 4),
                each Table.PromoteHeaders(Table.Transpose(_))
            )
        )
    in
      #"Split Transpose Combine"

     

    Personally, I'd recommend unpivoting the Staff ID too so that your final result looks like this:

    This format should be easier to work with if you plan to build a report off of it.

     

    Extra steps applied:

    #"Renamed Columns" = Table.RenameColumns(#"Split Transpose Combine",{{"Staff ID", "Date"}}),
    #"Unpivoted Other Columns" = Table.UnpivotOtherColumns(#"Renamed Columns", {"Date"}, "StaffID", "Value")

     

  • I started on this before I noticed the other responses, but here's one way to do it in the query editor.  To see how it works, just create a blank query, open the Advanced Editor and replace the text there with the M code below.

     

    let
        Source = Table.FromRows(Json.Document(Binary.Decompress(Binary.FromText("jZGxDoMwEEN/BWVGIr5Aku5dmDtGDF34gv6/eqkqMLobGCyesCL57NbC6/Pe92F9hjFgijJJlKQsxIl4Jl4O3sYWkGboz6KqqnxwN0tR2zidf08lZQhZsSc4Hg9KVt2h8JmCFeJK/CBG9NJHIy/9+b2kr0a304O7B5cPbh9cP9z+OUA2Fywk5wI+/Zzn3gU8AHgB8ATgDcTd4Jre28C67gZ/e/sC", BinaryEncoding.Base64), Compression.Deflate)), let _t = ((type nullable text) meta [Serialized.Text = true]) in type table [Column1 = _t, Column2 = _t, Column3 = _t, Column4 = _t, Column5 = _t, Column6 = _t]),
        #"Changed Type" = Table.TransformColumnTypes(Source,{{"Column1", type text}, {"Column2", type text}, {"Column3", type text}, {"Column4", type text}, {"Column5", type text}, {"Column6", type text}}),
        #"Filtered Rows" = Table.SelectRows(#"Changed Type", each ([Column1] <> " ")),
        Custom1 = Table.FromColumns({List.Repeat(#"Filtered Rows"[Column1], List.Count(Table.ColumnNames(#"Filtered Rows"))-1), List.Combine(List.RemoveFirstN(Table.ToColumns(#"Filtered Rows"), 1))}),
        #"Added Custom" = Table.AddColumn(Custom1, "Custom", each if Text.Contains([Column2], "/") then [Column2] else null),
        #"Filled Down" = Table.FillDown(#"Added Custom",{"Custom"}),
        #"Filtered Rows1" = Table.SelectRows(#"Filled Down", each ([Column1] <> "Staff ID")),
        #"Renamed Columns" = Table.RenameColumns(#"Filtered Rows1",{{"Column1", "StaffID"}, {"Column2", "Hours"}, {"Custom", "Date"}}),
        #"Changed Type1" = Table.TransformColumnTypes(#"Renamed Columns",{{"StaffID", Int64.Type}, {"Hours", Int64.Type}}),
        #"Changed Type with Locale" = Table.TransformColumnTypes(#"Changed Type1", {{"Date", type date}}, "en-GB")
    in
        #"Changed Type with Locale"

     

    Pat

4 Replies

  • ppm1's avatar
    ppm1
    Solution Sage

    I started on this before I noticed the other responses, but here's one way to do it in the query editor.  To see how it works, just create a blank query, open the Advanced Editor and replace the text there with the M code below.

     

    let
        Source = Table.FromRows(Json.Document(Binary.Decompress(Binary.FromText("jZGxDoMwEEN/BWVGIr5Aku5dmDtGDF34gv6/eqkqMLobGCyesCL57NbC6/Pe92F9hjFgijJJlKQsxIl4Jl4O3sYWkGboz6KqqnxwN0tR2zidf08lZQhZsSc4Hg9KVt2h8JmCFeJK/CBG9NJHIy/9+b2kr0a304O7B5cPbh9cP9z+OUA2Fywk5wI+/Zzn3gU8AHgB8ATgDcTd4Jre28C67gZ/e/sC", BinaryEncoding.Base64), Compression.Deflate)), let _t = ((type nullable text) meta [Serialized.Text = true]) in type table [Column1 = _t, Column2 = _t, Column3 = _t, Column4 = _t, Column5 = _t, Column6 = _t]),
        #"Changed Type" = Table.TransformColumnTypes(Source,{{"Column1", type text}, {"Column2", type text}, {"Column3", type text}, {"Column4", type text}, {"Column5", type text}, {"Column6", type text}}),
        #"Filtered Rows" = Table.SelectRows(#"Changed Type", each ([Column1] <> " ")),
        Custom1 = Table.FromColumns({List.Repeat(#"Filtered Rows"[Column1], List.Count(Table.ColumnNames(#"Filtered Rows"))-1), List.Combine(List.RemoveFirstN(Table.ToColumns(#"Filtered Rows"), 1))}),
        #"Added Custom" = Table.AddColumn(Custom1, "Custom", each if Text.Contains([Column2], "/") then [Column2] else null),
        #"Filled Down" = Table.FillDown(#"Added Custom",{"Custom"}),
        #"Filtered Rows1" = Table.SelectRows(#"Filled Down", each ([Column1] <> "Staff ID")),
        #"Renamed Columns" = Table.RenameColumns(#"Filtered Rows1",{{"Column1", "StaffID"}, {"Column2", "Hours"}, {"Custom", "Date"}}),
        #"Changed Type1" = Table.TransformColumnTypes(#"Renamed Columns",{{"StaffID", Int64.Type}, {"Hours", Int64.Type}}),
        #"Changed Type with Locale" = Table.TransformColumnTypes(#"Changed Type1", {{"Date", type date}}, "en-GB")
    in
        #"Changed Type with Locale"

     

    Pat

    • Ja12345's avatar
      Ja12345
      Regular Visitor

      Awesome, I'm so glad you still posted as this is the perfect solution for me, it lets me look through each step and see what's happening. It's also dynamic based on the amount of staff IDs in the table, which can be variable. THANK YOU!

  • First, let's see how we could handle this if we take the first four rows of the sample data. That is, start from this:

     

    From here, we can transpose the table and promote the headers to get precisely the shape you specified.

     

    Here's the code for this simplified example:

    let
        Source = Table.FromRows(Json.Document(Binary.Decompress(Binary.FromText("i45WCi5JTEtT8HRR0lEy1Dcw0jcyMDIGso2Q2MZIbBMktimcHasTrWRobGIIFDQHYgsgNoOzQZLm5kBpDBkQG6zVyNjM0AhJygDkAojmWAA=", BinaryEncoding.Base64), Compression.Deflate)), let _t = ((type nullable text) meta [Serialized.Text = true]) in type table [Column1 = _t, Column2 = _t, Column3 = _t, Column4 = _t, Column5 = _t, Column6 = _t]),
        #"Transposed Table" = Table.Transpose(Source),
        #"Promoted Headers" = Table.PromoteHeaders(#"Transposed Table", [PromoteAllScalars=true])
    in
        #"Promoted Headers"

     

    Now we just need to do the same thing for each subtable. The trick is to use Table.Split to split the table into a list of subtables, transform each table in the list just like above, and then combine the list of tables back into a single table.

     

    Here's the code for this:

    let
        Source = Table.FromRows(Json.Document(Binary.Decompress(Binary.FromText("jZGxDoMwEEN/BWVGIr5Aku5dmDtGDF34gv6/eqkqMLobGCyesCL57NbC6/Pe92F9hjFgijJJlKQsxIl4Jl4O3sYWkGboz6KqqnxwN0tR2zidf08lZQhZsSc4Hg9KVt2h8JmCFeJK/CBG9NJHIy/9+b2kr0a304O7B5cPbh9cP9z+OUA2Fywk5wI+/Zzn3gU8AHgB8ATgDcTd4Jre28C67gZ/e/sC", BinaryEncoding.Base64), Compression.Deflate)), let _t = ((type nullable text) meta [Serialized.Text = true]) in type table [Column1 = _t, Column2 = _t, Column3 = _t, Column4 = _t, Column5 = _t, Column6 = _t]),
        #"Filtered Rows" = Table.SelectRows(Source, each ([Column1] <> " ")), 
        #"Split Transpose Combine" = Table.Combine(
            List.Transform(
                Table.Split(#"Filtered Rows", 4),
                each Table.PromoteHeaders(Table.Transpose(_))
            )
        )
    in
      #"Split Transpose Combine"

     

    Personally, I'd recommend unpivoting the Staff ID too so that your final result looks like this:

    This format should be easier to work with if you plan to build a report off of it.

     

    Extra steps applied:

    #"Renamed Columns" = Table.RenameColumns(#"Split Transpose Combine",{{"Staff ID", "Date"}}),
    #"Unpivoted Other Columns" = Table.UnpivotOtherColumns(#"Renamed Columns", {"Date"}, "StaffID", "Value")

     

  • wdx223_Daniel's avatar
    wdx223_Daniel
    Community Champion

    import your data without headers, ie. the header names should be Column1, Column2,..... etc.

    then try this code

    let
        Source = Excel.CurrentWorkbook(){[Name="Table1"]}[Content],
        Custom1 = Table.Combine(Table.Group(Source,"Column1",{"n",each Table.UnpivotOtherColumns(Table.PromoteHeaders(Table.RemoveLastN(_,each List.Contains({null,""},Text.Trim(Text.From([Column1]))))),{"Staff ID"},"Date","Value")},0,(x,y)=>Byte.From(y="Staff ID"))[n]),
        Custom2 = Table.TransformColumns(Custom1,{{"Staff ID",Text.From},{"Date",each Date.From(_,"fr")}}),
        Custom3 = Table.Pivot(Custom2,List.Distinct(Custom2[Staff ID]),"Staff ID","Value")
    in
        Custom3