Forum Discussion

useazebra's avatar
useazebra
Advocate I
8 years ago
Solved

Expanding Multiple Data Tables with Unknown Column Names

I'm creating a query which the user can easily point to an alternate source without having the query break. All source tables are highly standardized EXCEPT for the column headers (go figure).    D...
Need Help
  • ImkeF's avatar
    ImkeF
    8 years ago

    True :)

    Your syntax would only work if you would reference a column that contains the column names already like here for example: https://www.mrexcel.com/forum/power-bi/952568-power-query-expand-all-columns.html

     

    But in your case you have to include the Table.ColumnNames function to retrieve the column names first. If you want to include it into one step, it could look like so:

     

    #"Expanded Data" = Table.ExpandTableColumn(#"Filtered Rows", "Data", List.Union(List.Transform(#"Filtered Rows"[Data]), each Table.ColumnNames(_)))
Anonymous's avatar
Anonymous
Not applicable
7 years ago

Thank you for this solution and it works perfectly fine if we use get data from excel but if we get data from Folder then this run too much slow. I have a folder with 1 excel ( just for testing i kept 1 but there will be many files) and when i applied this approach then it was too slow when checked the status bar then found that it was loading more than 500Mb of data where as my file size is only 1Mb. Can you please help me how to solve this ( i know we can use Table.Combine to achive but that doesn't serve the purpose).

ImkeF's avatar
ImkeF
Community Champion
7 years ago

Hi Anonymous ,

not sure I can follow what you're describing here. Please post your M-code so that I can follow.

 

  • Anonymous's avatar
    Anonymous
    Not applicable
    7 years ago

    Below code works fine if I load data from excel i.e. selecting excel Data.xlsx

     

    let
    Source = Excel.Workbook(File.Contents(""E:\Project\TestFiles\Data.xlsx"), null, true),
    #"Added Custom" = Table.AddColumn(Source, "PromoteHeader", each Table.PromoteHeaders([Data])),
    #"Removed Columns" = Table.RemoveColumns(#"Added Custom",{"Data"}),
    DistinctColumn = List.Union(List.Transform(Table.Column(#"Removed Columns",Table.ColumnNames(#"Removed Columns"){0}), each Table.ColumnNames(_))), 
        ExpandTable =Table.ExpandTableColumn(#"Removed Columns","PromoteHeader",DistinctColumn,DistinctColumn)
    in
        ExpandTable

     

    But it is too much slow at last line if I load data from folder( in status bar it shows loding more than 500Mb data where as i have only 1 file in folder with size 1Mb. I have tried using Table.Buffer as well but it also runs slow.

    let
    Source = Folder.Files("E:\Project\TestFiles"),
    Step1 = Table.SelectColumns(Source,{"Content"}),
    Step2 = Table.AddColumn(Step1, "Custom", each Excel.Workbook([Content])),
    Step3 = Table.SelectColumns(Step2, {"Custom"}),
    Step4 = Table.ExpandTableColumn(Step3, "Custom", {"Name", "Data"}, {"Name", "Data"}),
    Step5 = Table.RemoveColumns(Step5 ,{"Name"}),
    Step6  = Table.AddColumn(Step6, "PromoteHeader", each Table.PromoteHeaders([Data])),
    Step7 = Table.SelectColumns(Step6,{"PromoteHeader"}),
    DistinctColumn = List.Union(List.Transform(Table.Column(Step7,Table.ColumnNames(Step7){0}), each Table.ColumnNames(_))),
    Step8 =Table.ExpandTableColumn(Step7,"PromoteHeader",DistinctColumn,DistinctColumn)
    in
        Step8

     

    • ImkeF's avatar
      ImkeF
      Community Champion
      7 years ago

      Hi Anonymous ,

      your code doesn't work for me. Please check if you have added or removed some steps here before pasting into the thread.

       

      What are you trying to achieve?:

      1.  Dynamically retrieve the names of the first file and apply that to all files of the folder or
      2.  Dynamically retrieve the names of ALL files in the folder and append all tables (with respective null values for columns that are not present in all tables)
      3.  
      • Anonymous's avatar
        Anonymous
        Not applicable
        7 years ago

        Hi ImkeF 

         

        I have a folder with multiple .xls file with same structure, it will change in future. I am trying to append all as a single file and but I don't need all the columns. I have around 40 columns and as per business we need only 20 columns. So I reffered your method of expanding the column dynamically. Below code works but it is too much slow.

         

        let
    Source = Folder.Files("E:\Project\TestFiles"),
    Step1 = Table.SelectColumns(Source,{"Content"}),
    Step2 = Table.AddColumn(Step1, "Custom", each Excel.Workbook([Content])),
    Step3 = Table.SelectColumns(Step2, {"Custom"}),
    Step4 = Table.ExpandTableColumn(Step3, "Custom", {"Name", "Data"}, {"Name", "Data"}),
    Step5 = Table.RemoveColumns(Step4 ,{"Name"}),
    Step6  = Table.AddColumn(Step5, "PromoteHeader", each Table.PromoteHeaders([Data])),
    Step7 = Table.SelectColumns(Step6,{"PromoteHeader"}),
    DistinctColumn = List.Union(List.Transform(Table.Column(Step7,Table.ColumnNames(Step7){0}), each Table.ColumnNames(_))),
    ExpandTable = Table.ExpandTableColumn(Step7,"PromoteHeader",DistinctColumn,DistinctColumn)
in
    ExpandTable

        For testing i just had 1 .xls file of 1.5Mb in folder but it was too slow and in status bar it was showing loading more than 500Mb( clueless how ). So again I tried using single excel file with below code and it worked perfectly fine.

         

        let
    Source = Excel.Workbook(File.Contents("E:\Project\TestFiles\Data.xls"), null, true),
    #"Added Custom" = Table.AddColumn(Source, "PromoteHeader", each Table.PromoteHeaders([Data])),
    #"Removed Columns" = Table.SelectColumns(#"Added Custom",{"PromoteHeader"}),
    DistinctColumn = List.Union(List.Transform(Table.Column(#"Removed Columns",Table.ColumnNames(#"Removed Columns"){0}), each Table.ColumnNames(_))), 
    ExpandTable =Table.ExpandTableColumn(#"Removed Columns","PromoteHeader",DistinctColumn,DistinctColumn)
in
    ExpandTable

        Now my question is:- 1. Why it is too slow for the first code? I am mising anything.

                                            2. How can it be fixed.

         

        Hope the above explanation gives you idea what I am trying to achive.