Forum Discussion
Sum all individual Columns for large data sets
- 1 year ago
I think the simplest approach for you is:
- Leave your CV query as is, where it just focuses on querying data source files, parsing, combining, etc.
- Right-click on your CV query, select Reference
(this will create a new query where your first step, Source, is just pointing to CV) - On your new query (rename to whatever), open the 'Advanced Editor', replace all text with a solution from here, just be sure that your Source step is referring to your original query, CV
- Most likely, you will want to turn off load on CV, so only the new table in desired output is added to your model and/or sheet
lbendlin's solution should work for you IF your columns are data type text and the only value you want to sum across all columns is "1" (as text).
In case your CV table has number/integer data typed columns, you can instead use the below code, following same approach, which is: 1) unpivot all columns, 2) extract group text between _'s, 3) pivot values back and sum
let // Reference to the original table Source = CV, // Unpivot all columns. Column names go to the "Attribute" column, // and values go to the "Value" column. UnpivotCols = Table.UnpivotOtherColumns(Source, {}, "Attribute", "Value"), // Extract your group text/labels from between the first and second _ ExtractGroupText = Table.TransformColumns( UnpivotCols, {{"Attribute", each Text.BetweenDelimiters(_, "_", "_"), type text}} ), // Pivot the table back, summing all values under matching group labels PivotValueAndSum = Table.Pivot( ExtractGroupText, List.Distinct(ExtractGroupText[Attribute]), "Attribute", "Value", List.Sum ) in PivotValueAndSumHere is a quick gif of the steps I outlined at top. Note: this is in PBI Desktop, not Excel, so UI may be a little different, but everything should apply the same, mostly.
Please provide sample data that covers your issue or question completely, in a usable format (not as a screenshot).
Do not include sensitive information. Do not include anything that is unrelated to the issue or question.
Please show the expected outcome based on the sample data you provided.
Need help uploading data? https://community.fabric.microsoft.com/t5/Community-Blog/How-to-provide-sample-data-in-the-Power-BI-Forum/ba-p/963216
Want faster answers? https://community.fabric.microsoft.com/t5/Desktop/How-to-Get-Your-Question-Answered-Quickly/m-p/1447523
The data is now edited to match my data completely.
- lbendlin1 year agoSuper User
let Source = Table.FromRows(Json.Document(Binary.Decompress(Binary.FromText("i45WMlTSUcorzclBUIZQDObG6kRjkUYXQVKF3TyIWbEA", BinaryEncoding.Base64), Compression.Deflate)), let _t = ((type nullable text) meta [Serialized.Text = true]) in type table [abc_X_123.stat.energize = _t, abc_X_456.stat.energize = _t, abc_X_789.stat.energize = _t, abc_Y_123.stat.energize = _t, abc_Z_123.stat.energize = _t, abc_Z_456.stat.energize = _t]), #"Unpivoted Columns" = Table.UnpivotOtherColumns(Source, {}, "Attribute", "Value"), #"Split Column by Delimiter" = Table.SplitColumn(#"Unpivoted Columns", "Attribute", Splitter.SplitTextByDelimiter("_", QuoteStyle.Csv), {"Attribute.1", "Attribute.2", "Attribute.3"}), #"Removed Other Columns" = Table.SelectColumns(#"Split Column by Delimiter",{"Attribute.2", "Value"}), #"Filtered Rows" = Table.SelectRows(#"Removed Other Columns", each ([Value] = "1")), #"Changed Type" = Table.TransformColumnTypes(#"Filtered Rows",{{"Value", Int64.Type}}), #"Pivoted Column" = Table.Pivot(#"Changed Type", List.Distinct(#"Changed Type"[Attribute.2]), "Attribute.2", "Value", List.Sum) in #"Pivoted Column"How to use this code: Create a new Blank Query. Click on "Advanced Editor". Replace the code in the window with the code provided here. Click "Done". Once you examined the code, replace the entire Source step with your own source.
NOTE: Your expected outcome is an anti-pattern. This is Power BI, not Excel. Leave the pivoting to the visuals.
- Rick_S1371 year agoHelper I
I don't fully understand how to use source code in power query, but I have a question about these lines,
in type table [abc_X_123.stat.energize = _t, abc_X_456.stat.energize = _t, abc_X_789.stat.energize = _t, abc_Y_123.stat.energize = _t, abc_Z_123.stat.energize = _t, abc_Z_456.stat.energize = _t]),
are these names hardcoded in, because these are not the actual names, and even if they were I have 200+ columns I cannot use any hardcoded mames as it would take me much to long to type all them in by hand.
I will still attempt to use this code in the meantime.
- lbendlin1 year agoSuper User
That is just the meta data of the sample data you provided.
Create a new Blank Query. Click on "Advanced Editor". Replace the code in the window with the code provided here. Click "Done". Once you examined the code, replace the entire Source step with your own source.