Forum Discussion
need help - Power query or M code solution needed
Hello .. Any help for the below problem is highly appreciated
Please refer source data as given below
1) The source is an excel sheet as shown above with blocks of data seperated by a blank row
2) The blocks have different number of columns . There are some common columns across blocks (e.g Monitor_group). However the position of common columns can differ from block to block
3) I want to align all common columns in one single column of the output table . The uncommon columns can appear after the common columns
4) Important - The source exce sheet is an incremental one . Hence in the future there could be new blocks of data with new columns and/or different sequence of columns . The solution should be able to handle this
5) The first column "Monitor" is already aligned however the values MAY not not be unique .
7) The solution should retain the existing rows .. Just want the common columns aligned . Blank rows may be removed
6) A power query solution or a M code solution , both is fine
7) If possible please share the solution PBI file
Hi ashwinkolte,
I hope this information is helpful. Please let me know if you have any further questions or if you'd like to discuss this further. If this answers your question, please Accept it as a solution and give it a 'Kudos' so others can find it easily.
Thank you.
27 Replies
- jgeddesSuper User
GroupKind.Local should be able to help you here.
Create a placeholder column that indicates the blank rows as null else a common value.From there you can group on the placeholder column, choose no aggregation. You will need to add GroupKind.Local as the fifth variable in the Table.Group() function.
Filter out the null rows.
Promote the headers in the nested tables.
Remove the outer placeholder column.
Expand the desired columns.
let Source = #table( {"Column1", "Column2", "Column3", "Column4", "Column5", "Column6"}, { {"Monitor", "Usergroup", "Monitor_group", "Contact", "", ""}, {"Monitor1", "Usergroup1", "Monitor_group1", "Contact1", "", ""}, {"Monitor2", "Usergroup2", "Monitor_group2", "Contact2", "", ""}, {"","","","","",""}, {"Monitor", "Poller", "Contact", "Monitor_group", "Usergroup", ""}, {"Monitor3", "Poller3", "Contact3", "Monitor_group3", "Usergroup3", ""}, {"Monitor4", "Poller4", "Contact4", "Monitor_group4", "Usergroup4", ""}, {"","","","","",""}, {"Monitor", "Contact", "Usergroup", "Poller", "Monitor_group", "Portnumber"}, {"Monitor5", "Contact5", "Usergroup5", "Poller4", "Monitor_group5", "Port5"}, {"Monitor6", "Contact6", "Usergroup6", "Poller5", "Monitor_group6", "Port6"} } ), add_local_group_column = Table.AddColumn( Source, "placeholder", each if [Column1] = "" then null else "G", type text ), local_group = Table.Group( add_local_group_column, {"placeholder"}, { {"AllRows", each Table.RemoveColumns(_, {"placeholder"}), type table} }, GroupKind.Local ), remove_null_rows = Table.SelectRows( local_group, each ([placeholder] = "G") ), promote_nested_headers = Table.TransformColumns( remove_null_rows, { {"AllRows", each Table.PromoteHeaders(_, [PromoteAllScalars = true])} } ), remove_placeholder = Table.RemoveColumns( promote_nested_headers, {"placeholder"} ), expand_desired = Table.ExpandTableColumn( remove_placeholder, "AllRows", {"Monitor", "Usergroup", "Monitor_group", "Contact", "Poller", "Portnumber"} ) in expand_desiredHope this helps.
- ashwinkolteHelper III
HI jgeddes
First of all thank you so much for super fast response. Really appreciate that . I saw the PBIX file and the code . However I see that Column names are hardcoded . As I mentioned earlier the future blocks can have more columns (upto 25) and hence more common columns . Can the code be changed to perform all the column operations dynamically
- lbendlinSuper User
Please provide sample data that covers your issue or question completely, in a usable format (not as a screenshot).
Do not include sensitive information. Do not include anything that is unrelated to the issue or question.
Please show the expected outcome based on the sample data you provided.
Need help uploading data? https://community.fabric.microsoft.com/t5/Community-Blog/How-to-provide-sample-data-in-the-Power-BI-Forum/ba-p/963216
Want faster answers? https://community.fabric.microsoft.com/t5/Desktop/How-to-Get-Your-Question-Answered-Quickly/m-p/1447523
- v-kpoloju-msftCommunity Support
Hi ashwinkolte,
Thank you for reaching out to the Microsoft fabric community forum. Thank you jgeddes, and lbendlin, for your inputs on this issue.
After thoroughly reviewing the details you provided, I was able to reproduce the scenario, and it worked on my end. I have used it as sample data on my end and successfully implemented it.I am also including .pbix file for your better understanding, please have a look into it:
If this post helps, then please give us ‘Kudos’ and consider Accept it as a solution to help the other members find it more quickly.
Thank you for using Microsoft Community Forum.- ashwinkolteHelper III
First of all thanks for responding
I saw the PBIX file and M code . if I am not wrong this code is using hardcoded column names . What if there there come additional ones which would come in the future in the input ? Will this work ?
#"Replaced Value" = Table.ReplaceValue(#"Transposed Table","",null,Replacer.ReplaceValue,{"Column1"}),
#"Replaced Value1" = Table.ReplaceValue(#"Replaced Value","",null,Replacer.ReplaceValue,{"Column2"}),
#"Replaced Value2" = Table.ReplaceValue(#"Replaced Value1","",null,Replacer.ReplaceValue,{"Column3"}),
#"Replaced Value3" = Table.ReplaceValue(#"Replaced Value2","",null,Replacer.ReplaceValue,{"Column4"}),
#"Transposed Table1" = Table.Transpose(#"Replaced Value3"),
#"Renamed Columns" = Table.RenameColumns(#"Transposed Table1",{{"Column1", "Monitor"}, {"Column4", "Monitor_group"}, {"Column2", "Usergroup"}, {"Column3", "Poller"}, {"Column5", "Contact"}}),
#"Filled Down" = Table.FillDown(#"Renamed Columns",{"Poller", "Column6"}),
#"Reordered Columns" = Table.ReorderColumns(#"Filled Down",{"Monitor", "Monitor_group", "Usergroup", "Poller", "Contact", "Column6"}),
#"Renamed Columns1" = Table.RenameColumns(#"Reordered Columns",{{"Column6", "Portnumber"}}),
#"Reordered Columns1" = Table.ReorderColumns(#"Renamed Columns1",{"Monitor", "Monitor_group", "Usergroup", "Poller", "Portnumber", "Contact"})
in
#"Reordered Columns1"- v-kpoloju-msftCommunity Support
Hi ashwinkolte,
Thank you for your detailed observation. You are right to raise this concern.The current M code relies on hardcoded column names such as "Column1", "Column2", etc. This method can lead to issues if the input file structure changes in the future (e.g., new columns are added or the column order changes), as the transformation steps may not function as expected or may even cause errors.
To make the ReplaceValue step future proof and apply it across all columns dynamically (including any that may be added later), you can use the code below. It will replace empty strings ("") with null across the entire table without needing to update the column list manually:
ReplaceBlanksWithNulls = Table.ReplaceValue( #"Transposed Table", "", null, Replacer.ReplaceValue, Table.ColumnNames(#"Transposed Table") )
This ensures that any new columns added to the source data will automatically be included in the transformation no code changes needed.If this post helps, then please give us ‘Kudos’ and consider Accept it as a solution to help the other members find it more quickly.
Thank you for using Microsoft Community Forum.