Forum Discussion
Dynamically Change Column Type based on Column Name/Type from another table
- 4 years ago
Hi KNP ,
funny, my understanding of Anonymous s request is just about a topic that I intended to blog about for some time (but too busy currently). Done that in some customer projects as well lately and it works really good. I find it especially useful when working with dataflows.
But I'm attaching the file here.
The transformations are done using this function:(TableToBeTransformed as table, TableName as text) => let FilterMatchingTableFromMapping = Table.SelectRows(ColumnMapping, each [Table] = TableName), TablesColumnNames = Table.FromColumns({Table.ColumnNames(TableToBeTransformed)}), FilterRelevantColumns = Table.NestedJoin( FilterMatchingTableFromMapping, {"old"}, TablesColumnNames, {"Column1"}, "TablesColumnNames", JoinKind.Inner ), RenameColumns = Table.RenameColumns( TableToBeTransformed, List.Transform(Table.ToRecords(FilterRelevantColumns), each {_[old], _[new]}) ), #"Changed Type" = Table.TransformColumnTypes( RenameColumns, List.Transform( Table.ToRecords(FilterRelevantColumns), each {_[new], fnReplaceTypes(_[Type])} ) ) in #"Changed Type"Hope this is what you've been looking for.
Thank you for your reply and the attached PBIX.
This is almost exactly what I need, with one change, in that the fChangeColumnsFromReference needs to have a second parameter to filter the ColumnMapping table by TableName.
The use case is the incoming column name(s) may all be the same across multiple tables, but the rename value is different. An example, using the sample provided:
| Table1 | Table2 | Table3 | |||
| old | new | old | new | old | new |
| C | Country | C | Countries | C | Country of Residence |
In this case, there are 3 tables, each get imported with a column 'C', but the new name for 'C' on each table is different because the tables represent different things.
In your example PBIX, I adapted the ColumnMapping to include a TableName value, which represents which Table does that rename belong to. So far so good.
Then I edited the fChangeColumnsFromReference to include a second 'TableName' parameter. When invoking this function directly, it works correctly and I get the correct rename value depending on which TableName I specified. Still so far so good.
However the issue I have that I just can't seem to get my head around, is how to edit the Table query calling the function, to pass the TableName parameter in. I'm still trying to grasp how the function, which expects a parameter, doesn't have the parameter specified within the Table Query.
I'm assuming that in the TransformColumnNames function, it is effectively iterating over each column name and passing the column name to the fChangeColumnsFromReference function.
This is the only step that i'm now stuck on.
For whatever reason it's not letting me attach the revised PBIX, but here are the queries:
fChangeColumnsFromReference
(TableName as text, ColumnName as text) as text =>
let
Source =
if (List.Contains(Record.FieldNames(#sections[Section1]), "ColumnMappingTest")) = true then
ColumnMappingTest
else
null,
ColumnNewName = try
if List.Contains(Table.SelectRows(Source, each ([Table_Name] = TableName))[old], ColumnName) = true then
if Text.Trim(Table.SelectRows(Table.SelectRows(Source, each ([Table_Name] = TableName)), each ([old] = ColumnName)){0}[new]) = "" then
ColumnName
else
Table.SelectRows(Table.SelectRows(Source, each ([Table_Name] = TableName)), each ([old] = ColumnName)){0}[new]
else
ColumnName
otherwise
ColumnName
in
ColumnNewName
Example Table 3 (Country of Residence)
let
Source = Table.FromRows(Json.Document(Binary.Decompress(Binary.FromText("dc6xCoQwDAbgd+ksR2096y64+QTFIdhydakQM/j4l0Q4ELnlzz98JInRjFAhgWm4IGbKKN07HyzPhBz0gWKWJpoJoa75QXulmDSSyt/Sea/EXZx1bc/zKMRZ8K/zvh3C663b8PpB6JzPbd1v1A2hs3IbDlIo15cv", BinaryEncoding.Base64), Compression.Deflate)), let _t = ((type nullable text) meta [Serialized.Text = true]) in type table [C = _t, P = _t, S = _t, renameOther = _t, htasht = _t]),
ChangedType = Table.TransformColumnTypes(Source,{{"C", type text}, {"P", type text}, {"S", type number}}),
RenameColumns = Table.TransformColumnNames( ChangedType, fChangeColumnsFromReferenceTest("Table3", each _) )
in
RenameColumns
Sorry, I'm short on time, late Friday afternoon and wife is on my case coz we're heading out 🤣, but I'm really keen to solve this.
Yeah, that had me for a while too, I understand it the same as you've interpreted it.
I think the key to making a completely dynamic solution is to identify the table name of the current query, that's the part I'm stuck on at the moment.
It's interesting to see what this does
let
Source = #sections,
Section1 = Record.FieldNames(Source[Section1]),
#"Converted to Table" = Table.FromList(Section1, Splitter.SplitByNothing(), null, null, ExtraValues.Error)
in
#"Converted to Table"
Paste it into a blank query.
Reordering the tables in the query editor and refreshing this causes the order to change.
The part that needs to be updated is something like...
[this is faux code, not tested and won't work]
if List.Contains(Source[old], ColumnName) and List.Contains(Source[table], TableName) = true then
if Text.Trim(Table.SelectRows(Source, each ([old] = ColumnName) and [table] = Source[table]){0}[new]) = "" then
I will have another look the next chance I get.
Best of luck.
There's a chance that converting this solution is over complicating things.
Just in case they have time I will try and summon the wonderful ImkeF or MarcelBeug for their expertise. They probably already have a function for it.
- ImkeF4 years agoCommunity Champion
Hi KNP ,
funny, my understanding of Anonymous s request is just about a topic that I intended to blog about for some time (but too busy currently). Done that in some customer projects as well lately and it works really good. I find it especially useful when working with dataflows.
But I'm attaching the file here.
The transformations are done using this function:(TableToBeTransformed as table, TableName as text) => let FilterMatchingTableFromMapping = Table.SelectRows(ColumnMapping, each [Table] = TableName), TablesColumnNames = Table.FromColumns({Table.ColumnNames(TableToBeTransformed)}), FilterRelevantColumns = Table.NestedJoin( FilterMatchingTableFromMapping, {"old"}, TablesColumnNames, {"Column1"}, "TablesColumnNames", JoinKind.Inner ), RenameColumns = Table.RenameColumns( TableToBeTransformed, List.Transform(Table.ToRecords(FilterRelevantColumns), each {_[old], _[new]}) ), #"Changed Type" = Table.TransformColumnTypes( RenameColumns, List.Transform( Table.ToRecords(FilterRelevantColumns), each {_[new], fnReplaceTypes(_[Type])} ) ) in #"Changed Type"Hope this is what you've been looking for.
- Anonymous4 years agoNot applicable
Thank you both for your assistance on this. I can confirm that the solution provided by ImkeF does everything I was looking for!
There were 2 very small changes / tweaks I made:
- The fnReplaceTypes were missing Int64.Type and Currency.Type which was a straightforward add to the query and that works
- Modified the TableName filter slightly. In my mapping table, while most tables are listed and mapped, I also included an 'ALL' table, where the incoming column name and rename applies to every table. In my project, that was UTC Load Date (Refreshed Date) which is on every table, so rather than mapping it over and over, it filters for the TableName passed in as the parameter AND the ALL table and that works lovely as well
I've updated my project and so far so good. I just need to publish it to the PBI Service and make sure everything is working.
Thank you both again for your assistance, it is much appreciated.
- KNP4 years agoSuper User
Thanks ImkeF - you always make that look so simple. I look forward to the blog.
Question - can you see a way to make the TableName dynamic? e.g.
= fnTransformFromMappingTable( Source, "Table2" ) the "Table2" to populate based on the query that the function is invoked in.
Soheil's function uses
let Source = #sections, Section1 = Record.FieldNames(Source[Section1]) ... ...to identify if a table exists, I wonder, is it possible to use a variation of this (or something else) to identify the current query?