Forum Discussion
How to combine Folders
- 6 years ago
Sky571 - see if this helps. It is not the way I would ultimately go about this as it is more difficult to maintain. I tried it using the two table transformation approach with a Table.Combine at the end and that is much more managable long term with the same results. However, this is with a single query, plus all of the Combine magic that Power Query does.
let Source = Folder.Files("C:\Users\Ed Hansberry\OneDrive\Work Stuff\Power BI Forum Examples\Test Files"), #"Filtered Rows" = Table.SelectRows(Source, each Text.Contains([Folder Path], "Import")), #"Filtered Rows1" = Table.SelectRows(#"Filtered Rows", each ([Extension] = ".txt")), #"Filtered Hidden Files1" = Table.SelectRows(#"Filtered Rows1", each [Attributes]?[Hidden]? <> true), #"Invoke Custom Function Tab" = Table.AddColumn(#"Filtered Hidden Files1", "Transform File Tab", each #"Transform File Tab"([Content])), #"Invoke Custom Function Pipe" = Table.AddColumn(#"Invoke Custom Function Tab", "Transform File Pipe", each Table.PromoteHeaders(#"Transform File Pipe"([Content]), [PromoteAllScalars=true])), #"Added Correct Delimiter" = Table.AddColumn(#"Invoke Custom Function Pipe", "Correct Delimiter", each if Table.ColumnNames([Transform File Pipe]){1} = "JournalLib" then [Transform File Pipe] else [Transform File Tab]), #"Renamed Columns1" = Table.RenameColumns(#"Added Correct Delimiter", {"Name", "Source.Name"}), #"Removed Other Columns1" = Table.SelectColumns(#"Renamed Columns1",{"Source.Name", "Correct Delimiter"}), #"Expanded Correct Delimiter" = Table.ExpandTableColumn(#"Removed Other Columns1", "Correct Delimiter", {"JournalCode", "JournalLib", "EcritureNum", "EcritureDate", "CompteNum", "CompteLib", "CompAuxNum", "CompAuxLib", "PieceRef", "PieceDate", "EcritureLib", "Debit", "Credit", "EcritureLet", "DateLet", "ValidDate", "Montantdevise", "Idevise", "DateRglt", "ModeRglt", "NatOp", "IdClient"}, {"JournalCode", "JournalLib", "EcritureNum", "EcritureDate", "CompteNum", "CompteLib", "CompAuxNum", "CompAuxLib", "PieceRef", "PieceDate", "EcritureLib", "Debit", "Credit", "EcritureLet", "DateLet", "ValidDate", "Montantdevise", "Idevise", "DateRglt", "ModeRglt", "NatOp", "IdClient"}) in #"Expanded Correct Delimiter"This does not tell the whole story though, so rather than post all of the precedent queries and functions, take a look at my PBIX attached. You will need to change the source directories to yours of course for this to work.
It provides this. You'll need to manually apply the Change Type step after this, and get rid of the blank rows. Many of your sample files only had a header row with no detail rows. Those get loaded. Just filter out nulls in the JournalCode field.How to use M code provided in a blank query:
1) In Power Query, select New Source, then Blank Query
2) On the Home ribbon, select "Advanced Editor" button
3) Remove everything you see, then paste the M code I've given you in that box.
4) Press Done
5) See this article if you need help using this M code in your model.
Sky571 So...maybe but I'm not entirely certain I understand what you are asking. Are you saying that you want to combine all of the files in a single folder and have the query figure out which delimiter each file has or are you saying you want one Folder query that really reaches out to 2 different folders and makes the necessary correction?
Hello Greg_Deckler ,
Thank you for answering quickly.
I think the 1st option is what I would like : to combine all of the files in a single folder and have the query figure out which delimiter each file has.
But the 2nd option is good also if I do not want to refresh 2 times instead of 1 (if I have I refresh for the main Folder, which will reaches out the 2 sub-folders) - because i have another case with files with 2 different columns , si i need to make some transformations.
Can you briefly tell me how to do it please?
Thanks xx
- Greg_Deckler6 years agoCommunity Champion
Sky571 - Sorry, one other question while I try to work through how to do this, is there any other way to identify the 2 sets of files other than which folder they are in and their delimiter? In other words, do they follow different file naming conventions or have different file extensions?
- Sky5716 years agoAdvocate I
Thank for your question.
The files have the same extensions (.txt)
For their names : they begin by the same first letter as FEC, but after it changes according to the beginning and ending of a fiscal period :
example: FEC - 01-01-2015 - 31-12-2015 => they have between 18 columns and 22, with the same order and name header, except for 1 case (where i have 2 columns with different names+content, so i need to transform those in order to harmonize with my first example file)
So,
What is different is :
1- Delimiters : either " " , or " I "
2- Name and content of 2 columns, which requires transformation queries
Hope this explanation will help !
- Greg_Deckler6 years agoCommunity Champion
Sky571 - I don't quite have my example working yet, but here is what I am thinking.
Your main query in a Folder query basically looks like this:
let Source = Folder.Files("C:\temp\powerbi\csvs"), #"Filtered Hidden Files1" = Table.SelectRows(Source, each [Attributes]?[Hidden]? <> true), #"Invoke Custom Function1" = Table.AddColumn(#"Filtered Hidden Files1", "Transform File", each #"Transform File"([Content])), #"Renamed Columns1" = Table.RenameColumns(#"Invoke Custom Function1", {"Name", "Source.Name"}), #"Removed Other Columns1" = Table.SelectColumns(#"Renamed Columns1", {"Source.Name", "Transform File"}), #"Expanded Table Column1" = Table.ExpandTableColumn(#"Removed Other Columns1", "Transform File", Table.ColumnNames(#"Transform File"(#"Sample File"))), #"Changed Type" = Table.TransformColumnTypes(#"Expanded Table Column1",{{"Source.Name", type text}, {"column1", Int64.Type}, {"column2", Int64.Type}, {"column3", Int64.Type}, {"date", type date}}) in #"Changed Type"So, you could do this:
let Source1 = Folder.Files("C:\temp\powerbi\csvs"), Source2 = Folder.Files("C:\temp\powerbi\csvs1"), Source = Table.Append(Source1,Source2), #"Filtered Hidden Files1" = Table.SelectRows(Source, each [Attributes]?[Hidden]? <> true), #"Invoke Custom Function1" = Table.AddColumn(#"Filtered Hidden Files1", "Transform File", each #"Transform File"([Content])), #"Renamed Columns1" = Table.RenameColumns(#"Invoke Custom Function1", {"Name", "Source.Name"}), #"Removed Other Columns1" = Table.SelectColumns(#"Renamed Columns1", {"Source.Name", "Transform File"}), #"Expanded Table Column1" = Table.ExpandTableColumn(#"Removed Other Columns1", "Transform File", Table.ColumnNames(#"Transform File"(#"Sample File"))), #"Changed Type" = Table.TransformColumnTypes(#"Expanded Table Column1",{{"Source.Name", type text}, {"column1", Int64.Type}, {"column2", Int64.Type}, {"column3", Int64.Type}, {"date", type date}}) in #"Changed Type"That part gets all of your files being fed through the same process. Next you have your Transform Sample File function that looks like this:
let Source = Csv.Document(Parameter1,[Delimiter=",", Columns=4, Encoding=1252, QuoteStyle=QuoteStyle.None]), #"Promoted Headers" = Table.PromoteHeaders(Source, [PromoteAllScalars=true]) in #"Promoted Headers"So what I am thinking is that you add a Parameter2 to the Helper Queries folder and you set that when you call the function. And you can tell what delimiter to set for this parameter because of what folder your file is coming from, so the Transform Sample File would end up looking like:
let Source = Csv.Document(Parameter1,[Delimiter=Parameter2, Columns=4, Encoding=1252, QuoteStyle=QuoteStyle.None]), #"Promoted Headers" = Table.PromoteHeaders(Source, [PromoteAllScalars=true]) in #"Promoted Headers"As I said, I don't quite have it all working just yet but figured I'd explain what I am thinking and maybe you can figure it all out before I can or maybe ImkeF and edhans or someone better at Power Query than I can step in and wrap it up. But, seems definitely possible to me.
- Sky5716 years agoAdvocate I
Thank you so much for your explanation. I am going to test and tell you how it goes. Thank you so much for your time ! 🙂
- Sky5716 years agoAdvocate I
Sorry Greg_Deckler
how do you do to name Source 1 and Source 2 ?
Source1 = Folder.Files("C:\temp\powerbi\csvs"), Source2 = Folder.Files("C:\temp\powerbi\csvs1"), Source = Table.Append(Source1,Source2),Thanks