Forum Discussion
Load data from folder multiple CSV files different schema (Power Query) - Possible?
- 4 years ago
rodneyc8063_1 , if they have nothing in common with each other, unfortunately you'll have to transform them all separately. =(
rodneyc8063_1 , typically I would combine files that are like for like within the same folder. You could possibly explore the data and group them by specific folders.
However, if you know that in each of the CSV files you're picking up only the required fields, it's very possible to just have them dumped into one folder with the following transformation logic within the helper query:
1. Select all files in folder
2. Promote header. Remove fixed column evaluation at your source step
3. Select only required columns. Add optional MissingField.Ignore
let
// Remove fixed column criteria
Source = Csv.Document(Parameter1,[Delimiter=",", Encoding=1252, QuoteStyle=QuoteStyle.None]),
// Go ahead and promote all headers
PromoteHeaders = Table.PromoteHeaders(Source, [PromoteAllScalars=true]),
// Select columns. Ensure you add the optional MissingField.Ignore
SelectRequiredColumns = Table.SelectColumns(PromoteHeaders,{"Column1", "Column2", "Column3", "Column4"}, MissingField.Ignore)
in
SelectRequiredColumns
As you can see, in this sample, I've asked for 4 columns but since only 2 exists, only 2 is returned.
And when I expand to combine all the CSV files, in one file I have two additional columns and it'll pick it up as well.