Forum Discussion
Error handling in loading files from folder
- 8 years ago
I found a solution for this and wanted to share it:
I created a function to handle the load of the individual Excel sheet. I then use this funtion in to individual queries. One to load the data from the files and one that will return the file name of the files that are unable to be processed by the funtion.
The function look like this:
(sourceFile as text) as table => let Source = Excel.Workbook(File.Contents(sourceFile), null, true), ExpandData = Table.ExpandTableColumn( Source, "Data", {"Column1", "Column2", "Column3", "Column4", "Column5", "Column6", "Column7", "Column8", "Column9", "Column10", "Column11", "Column12", "Column13", "Column14", "Column15"}, {"Column1", "Column2", "Column3", "Column4", "Column5", "Column6", "Column7", "Column8", "Column9", "Column10", "Column11", "Column12", "Column13", "Column14", "Column15"} ), AddRowNumber = Table.AddIndexColumn(ExpandData, "Row", 0, 1), //Remove Rows with only Nulls AddColumnRowWithData = Table.AddColumn(AddRowNumber, "RowWithData", each if [Column1] = null and [Column2] = null and [Column3] = null and [Column4] = null and [Column5] = null and [Column6] = null and [Column7] = null and [Column8] = null and [Column9] = null and [Column10] = null and [Column11] = null and [Column12] = null and [Column13] = null and [Column14] = null and [Column15] = null then "Empty Row" else "Row with data" ), RemoveRowsWithNoData = Table.SelectRows(AddColumnRowWithData, each ([RowWithData] = "Row with data")), RemoveColomnRowWithData = Table.RemoveColumns(RemoveRowsWithNoData, {"RowWithData"}), //Remove Columns with No Name TransposeTable = Table.Transpose(RemoveColomnRowWithData), RemoveColumnsWithNoName = Table.SelectRows(TransposeTable , each ([Column1] <> null)), TransposeTable2 = Table.Transpose(RemoveColumnsWithNoName), PromoteHeaders = Table.PromoteHeaders(TransposeTable2, [PromoteAllScalars=true]), RenameColumn = Table.RenameColumns(PromoteHeaders,{{"0", "Row"}}) in RenameColumnThe code for returning a list of files with errors:
let FilePath = fnGetParameter("File Path"), Source = Folder.Files(FilePath), RemoveOtherColumns = Table.SelectColumns( Source, {"Name", "Folder Path"}), CallFunction = Table.AddColumn(RemoveOtherColumns, "LoadAllFiles", each fnLoadFiles([Folder Path]&[Name])), KeepFilesWithErrors = Table.SelectRowsWithErrors(CallFunction, {"LoadAllFiles"}), RemoveColumns = Table.RemoveColumns(KeepFilesWithErrors ,{"Folder Path", "LoadAllFiles"}) in RemoveColumns
Hi sdjensen,
For your scenario, I'd like to suggest you to use programming languages to deal with these data validation and error handling, then output the formatted data. (I think it will be more simply than power query)
Finally, you can use power bi to import these analysis/formatted data.
Regards,
Xiaxoin Sheng
I found a solution for this and wanted to share it:
I created a function to handle the load of the individual Excel sheet. I then use this funtion in to individual queries. One to load the data from the files and one that will return the file name of the files that are unable to be processed by the funtion.
The function look like this:
(sourceFile as text) as table =>
let
Source = Excel.Workbook(File.Contents(sourceFile), null, true),
ExpandData = Table.ExpandTableColumn(
Source,
"Data",
{"Column1", "Column2", "Column3", "Column4", "Column5", "Column6", "Column7", "Column8", "Column9", "Column10", "Column11", "Column12", "Column13", "Column14", "Column15"},
{"Column1", "Column2", "Column3", "Column4", "Column5", "Column6", "Column7", "Column8", "Column9", "Column10", "Column11", "Column12", "Column13", "Column14", "Column15"}
),
AddRowNumber = Table.AddIndexColumn(ExpandData, "Row", 0, 1),
//Remove Rows with only Nulls
AddColumnRowWithData = Table.AddColumn(AddRowNumber, "RowWithData", each
if [Column1] = null
and [Column2] = null
and [Column3] = null
and [Column4] = null
and [Column5] = null
and [Column6] = null
and [Column7] = null
and [Column8] = null
and [Column9] = null
and [Column10] = null
and [Column11] = null
and [Column12] = null
and [Column13] = null
and [Column14] = null
and [Column15] = null
then "Empty Row"
else "Row with data"
),
RemoveRowsWithNoData = Table.SelectRows(AddColumnRowWithData, each ([RowWithData] = "Row with data")),
RemoveColomnRowWithData = Table.RemoveColumns(RemoveRowsWithNoData, {"RowWithData"}),
//Remove Columns with No Name
TransposeTable = Table.Transpose(RemoveColomnRowWithData),
RemoveColumnsWithNoName = Table.SelectRows(TransposeTable , each ([Column1] <> null)),
TransposeTable2 = Table.Transpose(RemoveColumnsWithNoName),
PromoteHeaders = Table.PromoteHeaders(TransposeTable2, [PromoteAllScalars=true]),
RenameColumn = Table.RenameColumns(PromoteHeaders,{{"0", "Row"}})
in
RenameColumn
The code for returning a list of files with errors:
let
FilePath = fnGetParameter("File Path"),
Source = Folder.Files(FilePath),
RemoveOtherColumns = Table.SelectColumns( Source, {"Name", "Folder Path"}),
CallFunction = Table.AddColumn(RemoveOtherColumns, "LoadAllFiles", each fnLoadFiles([Folder Path]&[Name])),
KeepFilesWithErrors = Table.SelectRowsWithErrors(CallFunction, {"LoadAllFiles"}),
RemoveColumns = Table.RemoveColumns(KeepFilesWithErrors ,{"Folder Path", "LoadAllFiles"})
in
RemoveColumns - KCinMelbourne7 years agoResolver I
I know this is old but thanks for sharing the solution you found. I'm in the same scenario. You dont happen to have a file you can share where this function is operating? I use a lot more DAX than M so its a little difficult to follow.