Forum Discussion

ashwinkolte's avatar
ashwinkolte
Helper III
1 year ago
Solved

need help - Power query or M code solution needed

Hello .. Any help for the below problem is highly appreciated 

 

Please refer source data as given below 

 

 

1) The source is an excel sheet as shown above with blocks of data seperated by a blank row

2) The blocks have different number of columns . There are some common columns across blocks (e.g Monitor_group). However the position of common columns can differ from block to block

3) I want to align all common columns in one single column of the output table . The uncommon columns can appear after the common columns 

4) Important - The source exce sheet is an incremental one . Hence in the future there could be new blocks of data with new columns and/or different sequence of columns . The solution should be able to handle this 

5) The first column "Monitor" is already aligned however the values MAY not not be unique .

7) The solution should retain the existing rows .. Just want the common columns aligned . Blank rows may be removed 

6) A power query solution or a M code solution , both is fine

7) If possible please share the solution PBI file 

  • v-kpoloju-msft's avatar
    v-kpoloju-msft
    1 year ago

    Hi ashwinkolte,


    I hope this information is helpful. Please let me know if you have any further questions or if you'd like to discuss this further. If this answers your question, please Accept it as a solution and give it a 'Kudos' so others can find it easily.


    Thank you.

27 Replies

  • GroupKind.Local should be able to help you here.

    Create a placeholder column that indicates the blank rows as null else a common value.

    From there you can group on the placeholder column, choose no aggregation. You will need to add GroupKind.Local as the fifth variable in the Table.Group() function.

    Filter out the null rows.

    Promote the headers in the nested tables.

    Remove the outer placeholder column.

    Expand the desired columns.

    let
        Source = 
        #table(
            {"Column1", "Column2", "Column3", "Column4", "Column5", "Column6"},
            {
                {"Monitor", "Usergroup", "Monitor_group", "Contact", "", ""},
                {"Monitor1", "Usergroup1", "Monitor_group1", "Contact1", "", ""},
                {"Monitor2", "Usergroup2", "Monitor_group2", "Contact2", "", ""},
                {"","","","","",""},
                {"Monitor", "Poller", "Contact", "Monitor_group", "Usergroup", ""},
                {"Monitor3", "Poller3", "Contact3", "Monitor_group3", "Usergroup3", ""},
                {"Monitor4", "Poller4", "Contact4", "Monitor_group4", "Usergroup4", ""},
                {"","","","","",""},
                {"Monitor", "Contact", "Usergroup", "Poller", "Monitor_group", "Portnumber"},
                {"Monitor5", "Contact5", "Usergroup5", "Poller4", "Monitor_group5", "Port5"},
                {"Monitor6", "Contact6", "Usergroup6", "Poller5", "Monitor_group6", "Port6"}
            }
        ),
        add_local_group_column = 
        Table.AddColumn(
            Source, 
            "placeholder", 
            each if [Column1] = "" then null else "G", 
            type text
        ),
        local_group = 
        Table.Group(
            add_local_group_column, 
            {"placeholder"}, 
            {
                {"AllRows", each Table.RemoveColumns(_, {"placeholder"}), type table}
            }, 
            GroupKind.Local
        ),
        remove_null_rows = 
        Table.SelectRows(
            local_group, 
            each ([placeholder] = "G")
        ),
        promote_nested_headers = 
        Table.TransformColumns(
            remove_null_rows, 
            {
                {"AllRows", each Table.PromoteHeaders(_, [PromoteAllScalars = true])}
            }
        ),
        remove_placeholder = 
        Table.RemoveColumns(
            promote_nested_headers,
            {"placeholder"}
        ),
        expand_desired = 
        Table.ExpandTableColumn(
            remove_placeholder, 
            "AllRows", 
            {"Monitor", "Usergroup", "Monitor_group", "Contact", "Poller", "Portnumber"}
        )
    in
        expand_desired

      Hope this helps.

     

  • v-kpoloju-msft's avatar
    v-kpoloju-msft
    Community Support

    Hi ashwinkolte,
    Thank you for reaching out to the Microsoft fabric community forum. Thank you jgeddes, and lbendlin, for your inputs on this issue.


    After thoroughly reviewing the details you provided, I was able to reproduce the scenario, and it worked on my end. I have used it as sample data on my end and successfully implemented it.

    I am also including .pbix file for your better understanding, please have a look into it:

    If this post helps, then please give us ‘Kudos’ and consider Accept it as a solution to help the other members find it more quickly.

    Thank you for using Microsoft Community Forum.



    • ashwinkolte's avatar
      ashwinkolte
      Helper III

      Hi v-kpoloju-msft 

       

      First of all thanks for responding 

       

      I saw the PBIX file and M code . if I am not wrong this code is using hardcoded column names . What if there there come additional ones which would come in the future in the input ? Will this work ?

       

       

       

      #"Replaced Value" = Table.ReplaceValue(#"Transposed Table","",null,Replacer.ReplaceValue,{"Column1"}),
      #"Replaced Value1" = Table.ReplaceValue(#"Replaced Value","",null,Replacer.ReplaceValue,{"Column2"}),
      #"Replaced Value2" = Table.ReplaceValue(#"Replaced Value1","",null,Replacer.ReplaceValue,{"Column3"}),
      #"Replaced Value3" = Table.ReplaceValue(#"Replaced Value2","",null,Replacer.ReplaceValue,{"Column4"}),
      #"Transposed Table1" = Table.Transpose(#"Replaced Value3"),
      #"Renamed Columns" = Table.RenameColumns(#"Transposed Table1",{{"Column1", "Monitor"}, {"Column4", "Monitor_group"}, {"Column2", "Usergroup"}, {"Column3", "Poller"}, {"Column5", "Contact"}}),
      #"Filled Down" = Table.FillDown(#"Renamed Columns",{"Poller", "Column6"}),
      #"Reordered Columns" = Table.ReorderColumns(#"Filled Down",{"Monitor", "Monitor_group", "Usergroup", "Poller", "Contact", "Column6"}),
      #"Renamed Columns1" = Table.RenameColumns(#"Reordered Columns",{{"Column6", "Portnumber"}}),
      #"Reordered Columns1" = Table.ReorderColumns(#"Renamed Columns1",{"Monitor", "Monitor_group", "Usergroup", "Poller", "Portnumber", "Contact"})
      in
      #"Reordered Columns1"

      • v-kpoloju-msft's avatar
        v-kpoloju-msft
        Community Support

        Hi ashwinkolte,
        Thank you for your detailed observation. You are right to raise this concern.

        The current M code relies on hardcoded column names such as "Column1", "Column2", etc. This method can lead to issues if the input file structure changes in the future (e.g., new columns are added or the column order changes), as the transformation steps may not function as expected or may even cause errors.

        To make the ReplaceValue step future proof and apply it across all columns dynamically (including any that may be added later), you can use the code below. It will replace empty strings ("") with null across the entire table without needing to update the column list manually:

        ReplaceBlanksWithNulls = Table.ReplaceValue(
        
            #"Transposed Table",
        
            "",
        
            null,
        
            Replacer.ReplaceValue,
        
            Table.ColumnNames(#"Transposed Table")
        
        )


        This ensures that any new columns added to the source data will automatically be included in the transformation no code changes needed.

        If this post helps, then please give us ‘Kudos’ and consider Accept it as a solution to help the other members find it more quickly.

        Thank you for using Microsoft Community Forum.