Forum Discussion
combining Excel files with dfferent columns
- 3 years ago
Hello - yes, this is possible. You would just change the portion of the script that removes columns to only remove those that have duplicates, not to remove the empty columns.
BEFORE
// Combine duplicate plan columns and generic columns into one list. ColumnNamesToRemove = List.Combine ( { DuplicatePlanColumns, GenericColumnNames } ), // -------------------------------------------------------- // Clean up the column names. // -------------------------------------------------------- // Remove specific columns from the source table. RemoveColumns = Table.RemoveColumns ( Source, ColumnNamesToRemove ),AFTER
// Combine duplicate plan columns and generic columns into one list. // ColumnNamesToRemove = List.Combine ( { DuplicatePlanColumns, GenericColumnNames } ), // -------------------------------------------------------- // Clean up the column names. // -------------------------------------------------------- // Remove specific columns from the source table. RemoveColumns = Table.RemoveColumns ( Source, DuplicatePlanColumns),
Hello - another option would be to normalize the tables without a mapping table prior appending. This solution dynamically removes the years from the column names and places it in two normalized columns: Actuals Year and Forecast Year. I have included explanations for each step in the query script below.
let
Source = Table.FromRows(Json.Document(Binary.Decompress(Binary.FromText("i45WcgQCJR0lQyNjEyCFjAJyEvNAMgZAAKURFJIkjIqNBQA=", BinaryEncoding.Base64), Compression.Deflate)), let _t = ((type nullable text) meta [Serialized.Text = true]) in type table [#"Project Title" = _t, #"Project Number" = _t, Column1 = _t, Column2 = _t, Column3 = _t, Column4 = _t, #"2022 PLAN" = _t, #"2022 Budget" = _t, #"Jan Actual Spend (2022)" = _t, #"Feb Actual Spend (2022)" = _t, #"2023 PLAN" = _t, #"Jan Forecast Spend (2023)" = _t, #"Feb Forecast Spend (2023)" = _t]),
// --------------------------------------------------------
// Store some variables
// --------------------------------------------------------
// The column names actually appearing in the source file.
FileColumnNames = Table.ColumnNames ( Source ),
// The year appearing in the Actual Spend columns.
GetYear_Actual = Text.Select (
List.First (
List.Select (
FileColumnNames, each Text.Contains ( _, "Actual Spend", Comparer.OrdinalIgnoreCase )
)
), {"0".."9"}
),
// The year appearing in the Forecast Spend columns.
GetYear_Forecast = Text.Select (
List.First (
List.Select (
FileColumnNames, each Text.Contains ( _, "Forecast Spend", Comparer.OrdinalIgnoreCase )
)
), {"0".."9"}
),
// Get a list of column names including the word 'Plan' (case non-sensitive), excluding the first.
DuplicatePlanColumns = List.RemoveFirstN (
List.Select (
FileColumnNames, each Text.Contains ( _, "Plan", Comparer.OrdinalIgnoreCase )
)
),
// Get a list of column names that start with 'Column' (case non-sensitive).
GenericColumnNames = List.Select (
FileColumnNames, each Text.StartsWith ( _, "Column", Comparer.OrdinalIgnoreCase )
),
// Combine duplicate plan columns and generic columns into one list.
ColumnNamesToRemove = List.Combine ( { DuplicatePlanColumns, GenericColumnNames } ),
// --------------------------------------------------------
// Clean up the column names.
// --------------------------------------------------------
// Remove specific columns from the source table.
RemoveColumns = Table.RemoveColumns ( Source, ColumnNamesToRemove ),
// Remove numbers (years), parenthesis and leading/trailing spaces.
TransformedFileColumnNames = Table.TransformColumnNames (
RemoveColumns,
each Text.Trim ( Text.Remove ( _, {"0".."9", "(", ")"} ) )
),
// --------------------------------------------------------
// If they don't already exist, add years columns to the table.
// --------------------------------------------------------
AddYear_Actual =
if not Table.HasColumns ( TransformedFileColumnNames, {"Actual Spend"} )
then Table.AddColumn ( TransformedFileColumnNames, "Actual Year", each GetYear_Actual, Int64.Type )
else TransformedFileColumnNames,
AddYear_Forecast =
if not Table.HasColumns ( AddYear_Actual, {"Forecast Spend"} )
then Table.AddColumn ( AddYear_Actual, "Forecast Year", each GetYear_Forecast, Int64.Type )
else AddYear_Actual
in
AddYear_Forecast
You can put it in a function like this:
let
fn = ( Table as table ) =>
let
Source = Table,
// --------------------------------------------------------
// Store some variables
// --------------------------------------------------------
// The column names actually appearing in the source file.
FileColumnNames = Table.ColumnNames ( Source ),
// The year appearing in the Actual Spend columns.
GetYear_Actual = Text.Select (
List.First (
List.Select (
FileColumnNames, each Text.Contains ( _, "Actual Spend", Comparer.OrdinalIgnoreCase )
)
), {"0".."9"}
),
// The year appearing in the Forecast Spend columns.
GetYear_Forecast = Text.Select (
List.First (
List.Select (
FileColumnNames, each Text.Contains ( _, "Forecast Spend", Comparer.OrdinalIgnoreCase )
)
), {"0".."9"}
),
// Get a list of column names including the word 'Plan' (case non-sensitive), excluding the first.
DuplicatePlanColumns = List.RemoveFirstN (
List.Select (
FileColumnNames, each Text.Contains ( _, "Plan", Comparer.OrdinalIgnoreCase )
)
),
// Get a list of column names that start with 'Column' (case non-sensitive).
GenericColumnNames = List.Select (
FileColumnNames, each Text.StartsWith ( _, "Column", Comparer.OrdinalIgnoreCase )
),
// Combine duplicate plan columns and generic columns into one list.
ColumnNamesToRemove = List.Combine ( { DuplicatePlanColumns, GenericColumnNames } ),
// --------------------------------------------------------
// Clean up the column names.
// --------------------------------------------------------
// Remove specific columns from the source table.
RemoveColumns = Table.RemoveColumns ( Source, ColumnNamesToRemove ),
// Remove numbers (years), parenthesis and leading/trailing spaces.
TransformedFileColumnNames = Table.TransformColumnNames (
RemoveColumns,
each Text.Trim ( Text.Remove ( _, {"0".."9", "(", ")"} ) )
),
// --------------------------------------------------------
// If they don't already exist, add years columns to the table.
// --------------------------------------------------------
AddYear_Actual =
if not Table.HasColumns ( TransformedFileColumnNames, {"Actual Spend"} )
then Table.AddColumn ( TransformedFileColumnNames, "Actual Year", each GetYear_Actual, Int64.Type )
else TransformedFileColumnNames,
AddYear_Forecast =
if not Table.HasColumns ( AddYear_Actual, {"Forecast Spend"} )
then Table.AddColumn ( AddYear_Actual, "Forecast Year", each GetYear_Forecast, Int64.Type )
else AddYear_Actual
in
AddYear_Forecast
in
fn
Snip of Result
- ValeriaBreve3 years agoPost Partisan
jennratten , thanks! Very interesting approach. However, I would like to include all other columns as well - so practically column 9,10 and 11 belonging to the second file should be there in the final combination of files with null columns for the first file.
Is this at all possible?
Thanks again!
Kind regards
Valeria
- jennratten3 years agoSuper User
Hello - yes, this is possible. You would just change the portion of the script that removes columns to only remove those that have duplicates, not to remove the empty columns.
BEFORE
// Combine duplicate plan columns and generic columns into one list. ColumnNamesToRemove = List.Combine ( { DuplicatePlanColumns, GenericColumnNames } ), // -------------------------------------------------------- // Clean up the column names. // -------------------------------------------------------- // Remove specific columns from the source table. RemoveColumns = Table.RemoveColumns ( Source, ColumnNamesToRemove ),AFTER
// Combine duplicate plan columns and generic columns into one list. // ColumnNamesToRemove = List.Combine ( { DuplicatePlanColumns, GenericColumnNames } ), // -------------------------------------------------------- // Clean up the column names. // -------------------------------------------------------- // Remove specific columns from the source table. RemoveColumns = Table.RemoveColumns ( Source, DuplicatePlanColumns),- ValeriaBreve3 years agoPost Partisan
jennratten thanks! I am still testing and testing, I can't just get it to work yet as I want but I see the logic - so it's just a matter of keeping trying from my side 🙂
Thanks! 🙂