Forum Discussion

igonzalezb's avatar
igonzalezb
Helper I
6 years ago
Solved

Combining multiple csv files from folder, each with different number and/or order of columns

I have a folder with multiple csv files. For example: Location01.csv: Timestamp Flow Pressure1 1 10 35 2 12 35 Location02.csv: Timestamp Pressure1 Pressure2 1 51 24 ...
  • Anonymous's avatar
    Anonymous
    6 years ago

    an attempt at a quick and very rough solution

    The scenario:

    3 file csv in the directory: csv1,csv2,csv3

    function load in 4 file: the fourth raises error since doesn't exist.

     

     

    then the query concaTable put the tables together as required(?):

     

     

     

     

     

     

     

    in this form some parameters are hard coded, but it is not difficult to make them dynamic.
    I just wanted to propose an idea of how to set up a solution ..
    if interested, one can work to introduce the management of other aspects.

     

     

    the code:

    function readCsv:

     

     

    let
        csvNum=(m)=>
        let
        n=Text.From(m), 
        Source = Csv.Document(File.Contents("C:\Users\abcdefg\OneDrive - TIM\MyD2020\BI\loc0" & n & ".csv"),[Delimiter=";",  Encoding=65001, QuoteStyle=QuoteStyle.None]),
        csv = Table.PromoteHeaders(Source, [PromoteAllScalars=true])
        in Table.AddColumn(csv, "idx", each "csv"& n)
      
    in
       csvNum

     

     

     

    query concaTable:

     

     

     

    let
        Source = List.Transform({1..4}, each readCsvs(_)),
        cols=List.Union(List.Transform({1..4},each try Table.ColumnNames(Source{_-1}) otherwise {})),
        t= Table.FromRecords(List.Combine(List.Transform({0..2},each Table.ToRecords(Source{_}))),cols,MissingField.UseNull)
        
    in
        t

     

     

     

     

    #######edited########

    I have removed this line of code which was a step of a previous idea, but is no longer needed here

     

    head=Record.FromList(List.Repeat({""}, List.Count(cols)),cols ),