Forum Discussion

Charcho's avatar
Charcho
Helper I
1 year ago
Solved

New Columns with Specific Row Values from other Columns in Multiple Excel Files

Hello, I’m loading multiple Excel files from a folder in a PBI query. I want to fill a new column with the value 'Apple' (Column3-row1) and another new column with the value '2025' (Column3-row3). All of this needs to be done in the 'Transform Sample File' query so that it will work in all the Excel files added in the VVO2 query. Could anyone help me with this? Thank you

 

  • Charcho, check this

     

    Output (first few columns)

     

    Change folder addres in Source step to folder with your excel files

     

    let
        Source = Folder.Files("c:\Users\abc\Downloads\PowerQueryForum\Charcho\"),
        FilteredExtension = Table.SelectRows(Source, each ([Extension] = ".xlsx")),
        TransformedContent = Table.TransformColumns(FilteredExtension, {{"Content", each 
            [   a = Excel.Workbook(_){0}[Data],
                // T = BinaryToTable{0}[Content],
                T = a,
                R = [ Fruit = T{0}[Column1], Year = T{2}[Column1] ],
                RemovedTopRows = Table.PromoteHeaders(Table.Skip(T, each not List.Contains(Record.ToList(_), "jan", Comparer.OrdinalIgnoreCase))),
                FilteredActualYear = Table.FirstN(RemovedTopRows, each not List.Contains(Record.ToList(_), "jan", Comparer.OrdinalIgnoreCase)),
                FilteredOutDiffAndTotal = Table.SelectRows(FilteredActualYear, each not ([Column1] = null) and not (List.Contains(Record.ToList(_), {"total", "diff"}, (x,y)=> List.AnyTrue(List.Transform(y, (w)=> (Text.Contains(Text.From(x), w, Comparer.OrdinalIgnoreCase))))))),
                RemovedBlankColumns =
                    [ blankCols = Table.SelectRows(Table.TransformColumnTypes(Table.Profile(FilteredOutDiffAndTotal),{{"NullCount", Int64.Type}, {"Count", Int64.Type}}), each [Count] = [NullCount])[Column],
                      removed = Table.RemoveColumns(FilteredOutDiffAndTotal, blankCols)
                    ][removed],
                RenamedColumns = Table.RenameColumns(RemovedBlankColumns,{{"Column1", "Numbers"}}),
                MergedColumns = Table.CombineColumns(RenamedColumns, List.FirstN(List.Skip(Table.ColumnNames(RenamedColumns)), each Text.StartsWith(_, "Column")) ,Combiner.CombineTextByDelimiter("", QuoteStyle.None),"Titles"),
                RemovedSquareTitles = Table.TransformColumns(MergedColumns,{{"Titles", each Text.Trim(_, {"▪"}), type text}}),
                RemovedColumns = Table.SelectColumns(RemovedSquareTitles, List.Select(Table.ColumnNames(RemovedSquareTitles), each not Text.StartsWith(_, "Column"))),
                RenamedColumns2 = Table.RenameColumns(RemovedColumns,{{Table.ColumnNames(RemovedColumns){2}, "DecPrevious"}}),
                    RenamedColumns3 = Table.TransformColumnNames(RenamedColumns2, each Text.BeforeDelimiter(_, "_")),
                Ad_FuitAndYear = Table.FillDown(Table.FromColumns({{R[Fruit]}} & {{R[Year]}} & Table.ToColumns(RenamedColumns3), {"Fruit", "Year"} & Table.ColumnNames(RenamedColumns3)), {"Fruit", "Year"})
            ][Ad_FuitAndYear], type table}}),
        Combined = Table.Combine(TransformedContent[Content])
    
    in
        Combined

     

11 Replies

  • Anonymous's avatar
    Anonymous
    Not applicable

    Hi Charcho 
    The solution provided by dufoq3 is expected to work fine.please accept helpful reply as solution.This will be helpful for other community members who have similar problems to solve it faster.
    Thank you.

  • dufoq3's avatar
    dufoq3
    Community Champion

    Hi Charcho, check this:

     

    Before

     

    After

    let
        Source = Table.FromRows({{null, null, "apple"}, {null, null, "red"}, {null, null, 2025}}),
        Ad_AppleCol = Table.AddColumn(Source, "AppleColumn", each Source{0}[Column3], type text),
        Ad_YearCol = Table.AddColumn(Ad_AppleCol, "YearColumn", each Source{2}[Column3], Int64.Type)
    in
        Ad_YearCol

     

  • Thanks dufoq3 , however I would like a generic solution to apply the steps to all Excel files in the folder, with different fruits and years. The following picture is from another Excel file, where I would like 'Melon' and '2025' to be added in the same two new columns (with 'apple', ...)

    I have tried writing in the blank query ('Query1') the following, but I'm not sure how to proceed and integrate the solution into the final table 'VVO2' to apply the changes to all Excel files. is it ok? What should be the next step?

      

     

    • MarkLaf's avatar
      MarkLaf
      Super User

      Changes you make to the 'Transform Sample File' query automatically get duplicated in the 'Transform File' function, which is what gets used in the main combination query ('VVO2' in your case). That said, is there a reason not to add the transformations to 'VVO2' after the binaries are combined?

       

      Regardless of where you add the transformations, it's a little difficult to achieve your desired results because we need additional information to determine which text counts as valid fruit or which numbers count as valid years to put into their own respective columns. Is there info in another column that you can check to determine whether a given row in Column3 is of an applicable category?

       

      If not, you may have to just hard-code in what values should be applicable. Something like the below could work:

       

      let
          Source = Excel.Workbook(Parameter1, null, true),
          Sheet = Source{[Name="Sheet1"]}[Data],
      
          ValidFruit = {"Apple", "Melon", "Strawberry"},
          AddFruit = 
              Table.AddColumn(
                  Sheet, 
                  "Fruit", 
                  each if List.Contains( ValidFruit, [Column3] ) then [Column3] else null, 
                  type text 
              ),
      
          ValidYears = {2023, 2024, 2025},
          AddYear = 
              Table.AddColumn(
                  AddFruit, 
                  "Year", 
                  each if List.Contains( ValidYears, [Column3] ) then [Column3] else null, 
                  Int64.Type 
              )
      
      in
      
          AddYear

       

      Output:

       

      As an aside, in your latest snip, it looks like "Melon" is in your header, so you may want to consider an additional step somewhere that demotes headers (function: Table.DemoteHeaders / button: 'Use Headers as First Row')

    • dufoq3's avatar
      dufoq3
      Community Champion

      Hi, you do not need Transform File in general, but it is easier to manage just transform file for beginners. If you could upload at least 2 files (delete sensitive data in advance), we can try to help you.

    • dufoq3's avatar
      dufoq3
      Community Champion

      Charcho, try to replace whole your Transform Sample File code with this one:

       

      let
          Source = Excel.Workbook(Parameter1, null, true),
          Sheet = Source{[Name="Sheet1"]}[Data],
          F = (tbl as table)=>
              let col3 = Table.ColumnNames(tbl){2},
                  Ad_FruitCol = Table.AddColumn(tbl, "FruitColumn", each if col3 = "Column3" then tbl{0}[Column3] else col3, type text),
                  Ad_YearColumn = Table.AddColumn(Ad_FruitCol, "YearColumn", each Int64.From(if col3 = "Column3" then tbl{2}[Column3] else Record.Field(tbl{1}, col3)), Int64.Type)
              in  Ad_YearColumn,
          Ad_FruitAndYearCols = F(Sheet)
      in
          Ad_FruitAndYearCols

       

  • Anonymous's avatar
    Anonymous
    Not applicable

    Hi Charcho 

    Thank you for reaching out microsoft fabric community forum.

    May I ask if you have resolved this issue? If so, please mark the helpful reply and accept it as the solution. This will be helpful for other community members who have similar problems to solve it faster.

    Thank you.

  • Hello, 

    Thank you for the answers dufoq3 MarkLaf . The thing is, I have a lot of Excel sheets with specific data in the titles in half of them, and I need to combine them. Creating those two columns with the fruit and the year would be the first step. This should be done before merging the files, right?

    I've attached two example Excel files for reference and expected solution. https://www.dropbox.com/scl/fo/9qzy7t1nti2hgt0b2bsnr/AOXdgYX-n0wpPpMvDubjmVk?rlkey=z9g1t839bvk569oyxi550pqdr&st=yt3w5zll&dl=0

  • dufoq3's avatar
    dufoq3
    Community Champion

    Charcho, check this

     

    Output (first few columns)

     

    Change folder addres in Source step to folder with your excel files

     

    let
        Source = Folder.Files("c:\Users\abc\Downloads\PowerQueryForum\Charcho\"),
        FilteredExtension = Table.SelectRows(Source, each ([Extension] = ".xlsx")),
        TransformedContent = Table.TransformColumns(FilteredExtension, {{"Content", each 
            [   a = Excel.Workbook(_){0}[Data],
                // T = BinaryToTable{0}[Content],
                T = a,
                R = [ Fruit = T{0}[Column1], Year = T{2}[Column1] ],
                RemovedTopRows = Table.PromoteHeaders(Table.Skip(T, each not List.Contains(Record.ToList(_), "jan", Comparer.OrdinalIgnoreCase))),
                FilteredActualYear = Table.FirstN(RemovedTopRows, each not List.Contains(Record.ToList(_), "jan", Comparer.OrdinalIgnoreCase)),
                FilteredOutDiffAndTotal = Table.SelectRows(FilteredActualYear, each not ([Column1] = null) and not (List.Contains(Record.ToList(_), {"total", "diff"}, (x,y)=> List.AnyTrue(List.Transform(y, (w)=> (Text.Contains(Text.From(x), w, Comparer.OrdinalIgnoreCase))))))),
                RemovedBlankColumns =
                    [ blankCols = Table.SelectRows(Table.TransformColumnTypes(Table.Profile(FilteredOutDiffAndTotal),{{"NullCount", Int64.Type}, {"Count", Int64.Type}}), each [Count] = [NullCount])[Column],
                      removed = Table.RemoveColumns(FilteredOutDiffAndTotal, blankCols)
                    ][removed],
                RenamedColumns = Table.RenameColumns(RemovedBlankColumns,{{"Column1", "Numbers"}}),
                MergedColumns = Table.CombineColumns(RenamedColumns, List.FirstN(List.Skip(Table.ColumnNames(RenamedColumns)), each Text.StartsWith(_, "Column")) ,Combiner.CombineTextByDelimiter("", QuoteStyle.None),"Titles"),
                RemovedSquareTitles = Table.TransformColumns(MergedColumns,{{"Titles", each Text.Trim(_, {"▪"}), type text}}),
                RemovedColumns = Table.SelectColumns(RemovedSquareTitles, List.Select(Table.ColumnNames(RemovedSquareTitles), each not Text.StartsWith(_, "Column"))),
                RenamedColumns2 = Table.RenameColumns(RemovedColumns,{{Table.ColumnNames(RemovedColumns){2}, "DecPrevious"}}),
                    RenamedColumns3 = Table.TransformColumnNames(RenamedColumns2, each Text.BeforeDelimiter(_, "_")),
                Ad_FuitAndYear = Table.FillDown(Table.FromColumns({{R[Fruit]}} & {{R[Year]}} & Table.ToColumns(RenamedColumns3), {"Fruit", "Year"} & Table.ColumnNames(RenamedColumns3)), {"Fruit", "Year"})
            ][Ad_FuitAndYear], type table}}),
        Combined = Table.Combine(TransformedContent[Content])
    
    in
        Combined

     

  • Thank you very much dufoq3! it works in the example perfectly, but I am finding some problems when trying to adapt it to the real scenario. Would it be possible to simplify the code to only include the modification for adding two fixed new columns: "fruit" and "year"? 

    • dufoq3's avatar
      dufoq3
      Community Champion

      Sorry, but you should think of all possible scenarios in advance and provide comprehensive sample data with expected result. I have provided 3 different codes, as per your requirements, which you have changed over time