Forum Discussion
ValeriaBreve
3 years agoPost Partisan
combining Excel files with dfferent columns
Hello! I am trying to combine Excel files in Powerquery that have different columns: The year will always change, and that bit I was able to somehow capture dynamically based on the column position...
- 3 years ago
Hello - yes, this is possible. You would just change the portion of the script that removes columns to only remove those that have duplicates, not to remove the empty columns.
BEFORE
// Combine duplicate plan columns and generic columns into one list. ColumnNamesToRemove = List.Combine ( { DuplicatePlanColumns, GenericColumnNames } ), // -------------------------------------------------------- // Clean up the column names. // -------------------------------------------------------- // Remove specific columns from the source table. RemoveColumns = Table.RemoveColumns ( Source, ColumnNamesToRemove ),AFTER
// Combine duplicate plan columns and generic columns into one list. // ColumnNamesToRemove = List.Combine ( { DuplicatePlanColumns, GenericColumnNames } ), // -------------------------------------------------------- // Clean up the column names. // -------------------------------------------------------- // Remove specific columns from the source table. RemoveColumns = Table.RemoveColumns ( Source, DuplicatePlanColumns),
jennratten
3 years agoSuper User
Hello - another option would be to normalize the tables without a mapping table prior appending. This solution dynamically removes the years from the column names and places it in two normalized columns: Actuals Year and Forecast Year. I have included explanations for each step in the query script below.
let
Source = Table.FromRows(Json.Document(Binary.Decompress(Binary.FromText("i45WcgQCJR0lQyNjEyCFjAJyEvNAMgZAAKURFJIkjIqNBQA=", BinaryEncoding.Base64), Compression.Deflate)), let _t = ((type nullable text) meta [Serialized.Text = true]) in type table [#"Project Title" = _t, #"Project Number" = _t, Column1 = _t, Column2 = _t, Column3 = _t, Column4 = _t, #"2022 PLAN" = _t, #"2022 Budget" = _t, #"Jan Actual Spend (2022)" = _t, #"Feb Actual Spend (2022)" = _t, #"2023 PLAN" = _t, #"Jan Forecast Spend (2023)" = _t, #"Feb Forecast Spend (2023)" = _t]),
// --------------------------------------------------------
// Store some variables
// --------------------------------------------------------
// The column names actually appearing in the source file.
FileColumnNames = Table.ColumnNames ( Source ),
// The year appearing in the Actual Spend columns.
GetYear_Actual = Text.Select (
List.First (
List.Select (
FileColumnNames, each Text.Contains ( _, "Actual Spend", Comparer.OrdinalIgnoreCase )
)
), {"0".."9"}
),
// The year appearing in the Forecast Spend columns.
GetYear_Forecast = Text.Select (
List.First (
List.Select (
FileColumnNames, each Text.Contains ( _, "Forecast Spend", Comparer.OrdinalIgnoreCase )
)
), {"0".."9"}
),
// Get a list of column names including the word 'Plan' (case non-sensitive), excluding the first.
DuplicatePlanColumns = List.RemoveFirstN (
List.Select (
FileColumnNames, each Text.Contains ( _, "Plan", Comparer.OrdinalIgnoreCase )
)
),
// Get a list of column names that start with 'Column' (case non-sensitive).
GenericColumnNames = List.Select (
FileColumnNames, each Text.StartsWith ( _, "Column", Comparer.OrdinalIgnoreCase )
),
// Combine duplicate plan columns and generic columns into one list.
ColumnNamesToRemove = List.Combine ( { DuplicatePlanColumns, GenericColumnNames } ),
// --------------------------------------------------------
// Clean up the column names.
// --------------------------------------------------------
// Remove specific columns from the source table.
RemoveColumns = Table.RemoveColumns ( Source, ColumnNamesToRemove ),
// Remove numbers (years), parenthesis and leading/trailing spaces.
TransformedFileColumnNames = Table.TransformColumnNames (
RemoveColumns,
each Text.Trim ( Text.Remove ( _, {"0".."9", "(", ")"} ) )
),
// --------------------------------------------------------
// If they don't already exist, add years columns to the table.
// --------------------------------------------------------
AddYear_Actual =
if not Table.HasColumns ( TransformedFileColumnNames, {"Actual Spend"} )
then Table.AddColumn ( TransformedFileColumnNames, "Actual Year", each GetYear_Actual, Int64.Type )
else TransformedFileColumnNames,
AddYear_Forecast =
if not Table.HasColumns ( AddYear_Actual, {"Forecast Spend"} )
then Table.AddColumn ( AddYear_Actual, "Forecast Year", each GetYear_Forecast, Int64.Type )
else AddYear_Actual
in
AddYear_Forecast
You can put it in a function like this:
let
fn = ( Table as table ) =>
let
Source = Table,
// --------------------------------------------------------
// Store some variables
// --------------------------------------------------------
// The column names actually appearing in the source file.
FileColumnNames = Table.ColumnNames ( Source ),
// The year appearing in the Actual Spend columns.
GetYear_Actual = Text.Select (
List.First (
List.Select (
FileColumnNames, each Text.Contains ( _, "Actual Spend", Comparer.OrdinalIgnoreCase )
)
), {"0".."9"}
),
// The year appearing in the Forecast Spend columns.
GetYear_Forecast = Text.Select (
List.First (
List.Select (
FileColumnNames, each Text.Contains ( _, "Forecast Spend", Comparer.OrdinalIgnoreCase )
)
), {"0".."9"}
),
// Get a list of column names including the word 'Plan' (case non-sensitive), excluding the first.
DuplicatePlanColumns = List.RemoveFirstN (
List.Select (
FileColumnNames, each Text.Contains ( _, "Plan", Comparer.OrdinalIgnoreCase )
)
),
// Get a list of column names that start with 'Column' (case non-sensitive).
GenericColumnNames = List.Select (
FileColumnNames, each Text.StartsWith ( _, "Column", Comparer.OrdinalIgnoreCase )
),
// Combine duplicate plan columns and generic columns into one list.
ColumnNamesToRemove = List.Combine ( { DuplicatePlanColumns, GenericColumnNames } ),
// --------------------------------------------------------
// Clean up the column names.
// --------------------------------------------------------
// Remove specific columns from the source table.
RemoveColumns = Table.RemoveColumns ( Source, ColumnNamesToRemove ),
// Remove numbers (years), parenthesis and leading/trailing spaces.
TransformedFileColumnNames = Table.TransformColumnNames (
RemoveColumns,
each Text.Trim ( Text.Remove ( _, {"0".."9", "(", ")"} ) )
),
// --------------------------------------------------------
// If they don't already exist, add years columns to the table.
// --------------------------------------------------------
AddYear_Actual =
if not Table.HasColumns ( TransformedFileColumnNames, {"Actual Spend"} )
then Table.AddColumn ( TransformedFileColumnNames, "Actual Year", each GetYear_Actual, Int64.Type )
else TransformedFileColumnNames,
AddYear_Forecast =
if not Table.HasColumns ( AddYear_Actual, {"Forecast Spend"} )
then Table.AddColumn ( AddYear_Actual, "Forecast Year", each GetYear_Forecast, Int64.Type )
else AddYear_Actual
in
AddYear_Forecast
in
fn
Snip of Result