Forum Discussion
Iterate through folder and subfolders to get cvs data
Is it possible to iterate through a folder and subfolders as below, and combine targeted csv files into one table?
for example, I want to get all data (from day 1 to day 30) in one Folder_Jan2019, then iterate through other months folders as well, and combine them into a table. Possibly also iterate through Folder_2019, Folder_2020,etc. and combine all data from years as well.
If all of the files in the different subfolders have the same structure, you can use Folder.Files, and then the transform query will help you combine them. If there are differently structured files in each folder, I would replace Folder.Files with Folder.Contents, which will give you a view of the different folders, into which you can drill down into the individual files.
--Nate
2 Replies
If all of the files in the different subfolders have the same structure, you can use Folder.Files, and then the transform query will help you combine them. If there are differently structured files in each folder, I would replace Folder.Files with Folder.Contents, which will give you a view of the different folders, into which you can drill down into the individual files.
--Nate
Unless you construct some user-defined functions in PQ to traverse all folders and sub-folders, there's no functions out-of-the-box for such a task.
Powershell script comes to play
Get-ChildItem 'D:' -Recurse | Select-Object -Property fullname | ? {$_.FullName -match '\.csv$'} | Out-File 'E:\allcsv.csv'then import the summary file "E:\allcsv.csv" into PQ,
