Forum Discussion
Excel with different Schema (Missing Columns Errors)
Hello,
I am trying to combine multiple excel files in a folder. I was able to remove the trash rows, promote the headers (in transform sample file) in transform/query. Then remove the change data type and use "detect data type" to get rid of an error.
Unknown to me until now. There are some schema issues in the file. Some columns names are present in one file, but not there other. Familar with FieldsMissing.Ignore. How do I ignore the schema issues and bring in columns that are shared among the two?
New users. Thanks for any pointers.
Hi gordgord1, the correct approach depends on how you're using that/those missing column(s).
If you don't need the column in your report, you can set the file where the column is missing as the Sample File that is used as the "master" defining all the columns you expect in the other files too.You can do that either during the "Combine Files" step if you select "Combine & Transform" or by adjusting the Sample File in the Helper Queries:
If you do need the column but can accept that you'll have some null values, you can select the file that includes the column(s) as your Sample File instead. In that case the column will contain null-values for all rows of the sources that didn't contain the column.
If you are performing the same transformations to each table in the list of tables and you are selecting the same columns from each table you could use the Table.TransformColumns function in place of the stock UI combination.
If you have a starting point similar to...TableA
TableB
The following code will skip the first 5 rows, promote the headers and select the columns you indicated.
Table.TransformColumns( Source, { { "Data", each Table.SelectColumns( Table.PromoteHeaders( Table.Skip(_, 5) ), {"keepA", "keepB", "keepC", "keepD", "keepE"}, MissingField.Ignore ), type table [keepA=nullable text, keepB=nullable text, keepC=nullable text, keepD=nullable text, keepE=nullable text] } } )Once the tables are combined the result looks like...
I have attached the pbix for you to inspect as well.
This video solved my problem. How to use Power Query to Combine Multiple Files that have different headings by Access Analytic.
https://www.youtube.com/watch?v=09tvia_8ykI&t=1s
I had to further process the files and remove rows and promote headers in the "sample transform file". It was not working until I deleted the "changed type" that was added. Completely fixed my problem.
7 Replies
- KarinSzilagyiSuper User
Hi gordgord1, the correct approach depends on how you're using that/those missing column(s).
If you don't need the column in your report, you can set the file where the column is missing as the Sample File that is used as the "master" defining all the columns you expect in the other files too.You can do that either during the "Combine Files" step if you select "Combine & Transform" or by adjusting the Sample File in the Helper Queries:
If you do need the column but can accept that you'll have some null values, you can select the file that includes the column(s) as your Sample File instead. In that case the column will contain null-values for all rows of the sources that didn't contain the column.
- jgeddesSuper User
If you are performing the same transformations to each table in the list of tables and you are selecting the same columns from each table you could use the Table.TransformColumns function in place of the stock UI combination.
If you have a starting point similar to...TableA
TableB
The following code will skip the first 5 rows, promote the headers and select the columns you indicated.
Table.TransformColumns( Source, { { "Data", each Table.SelectColumns( Table.PromoteHeaders( Table.Skip(_, 5) ), {"keepA", "keepB", "keepC", "keepD", "keepE"}, MissingField.Ignore ), type table [keepA=nullable text, keepB=nullable text, keepC=nullable text, keepD=nullable text, keepE=nullable text] } } )Once the tables are combined the result looks like...
I have attached the pbix for you to inspect as well.
- gordgord1Regular Visitor
KarinSzilagyi Thanks. I do not need the columns causing the problems. I was not editing in the "sample file" area, but in "transform sample file." Thank you for pointing me in the right direction. WIll give a go tomorrow. After 3-4 hours of work, I need a mental break. 🙂
- KarinSzilagyiSuper User
gordgord1 Enjoy your mental detanglement, and feel free to ping me if you need additional help with this issue 😊
- v-pnaroju-msftCommunity Support
Thankyou, KarinSzilagyi and jgeddesfor your responses.
Hi gordgord1,
We appreciate your inquiry through the Microsoft Fabric Community Forum.
We would like to inquire whether have you got the chance to check the solutions provided by KarinSzilagyi and jgeddesto resolve the issue. We hope the information provided helps to clear the query. Should you have any further queries, kindly feel free to contact the Microsoft Fabric community.
Thank you. - v-pnaroju-msftCommunity Support
Hi gordgord1,
We are following up to see if what we shared solved your issue. If you need more support, please reach out to the Microsoft Fabric community.
Thank you. - gordgord1Regular Visitor
This video solved my problem. How to use Power Query to Combine Multiple Files that have different headings by Access Analytic.
https://www.youtube.com/watch?v=09tvia_8ykI&t=1s
I had to further process the files and remove rows and promote headers in the "sample transform file". It was not working until I deleted the "changed type" that was added. Completely fixed my problem.