Forum Discussion
Combining multiple csv files from folder, each with different number and/or order of columns
- Anonymous6 years ago
an attempt at a quick and very rough solution
The scenario:
3 file csv in the directory: csv1,csv2,csv3
function load in 4 file: the fourth raises error since doesn't exist.
then the query concaTable put the tables together as required(?):
in this form some parameters are hard coded, but it is not difficult to make them dynamic.
I just wanted to propose an idea of how to set up a solution .. if interested, one can work to introduce the management of other aspects.the code:
function readCsv:
let csvNum=(m)=> let n=Text.From(m), Source = Csv.Document(File.Contents("C:\Users\abcdefg\OneDrive - TIM\MyD2020\BI\loc0" & n & ".csv"),[Delimiter=";", Encoding=65001, QuoteStyle=QuoteStyle.None]), csv = Table.PromoteHeaders(Source, [PromoteAllScalars=true]) in Table.AddColumn(csv, "idx", each "csv"& n) in csvNumquery concaTable:
let Source = List.Transform({1..4}, each readCsvs(_)), cols=List.Union(List.Transform({1..4},each try Table.ColumnNames(Source{_-1}) otherwise {})), t= Table.FromRecords(List.Combine(List.Transform({0..2},each Table.ToRecords(Source{_}))),cols,MissingField.UseNull) in t#######edited########
I have removed this line of code which was a step of a previous idea, but is no longer needed here
head=Record.FromList(List.Repeat({""}, List.Count(cols)),cols ),
an attempt at a quick and very rough solution
The scenario:
3 file csv in the directory: csv1,csv2,csv3
function load in 4 file: the fourth raises error since doesn't exist.
then the query concaTable put the tables together as required(?):
in this form some parameters are hard coded, but it is not difficult to make them dynamic.
I just wanted to propose an idea of how to set up a solution .. if interested, one can work to introduce the management of other aspects.
the code:
function readCsv:
let
csvNum=(m)=>
let
n=Text.From(m),
Source = Csv.Document(File.Contents("C:\Users\abcdefg\OneDrive - TIM\MyD2020\BI\loc0" & n & ".csv"),[Delimiter=";", Encoding=65001, QuoteStyle=QuoteStyle.None]),
csv = Table.PromoteHeaders(Source, [PromoteAllScalars=true])
in Table.AddColumn(csv, "idx", each "csv"& n)
in
csvNum
query concaTable:
let
Source = List.Transform({1..4}, each readCsvs(_)),
cols=List.Union(List.Transform({1..4},each try Table.ColumnNames(Source{_-1}) otherwise {})),
t= Table.FromRecords(List.Combine(List.Transform({0..2},each Table.ToRecords(Source{_}))),cols,MissingField.UseNull)
in
t
#######edited########
I have removed this line of code which was a step of a previous idea, but is no longer needed here
head=Record.FromList(List.Repeat({""}, List.Count(cols)),cols ),
Glad you found a solution that works for you igonzalezb