Dealing with multiple excel file submission and data inconsistency

[osabhaois]

Hi

I am updating a data set on a weekly basis with multiple excel files. Some files are added to a folder and some are connected via onedrive web link.

The files are provided to me by team members and naturally sometimes columns can be missing.

When this happens to the OneDrive web linked files; this throws exception errors related to the missing columns.

I find that when I get data from a folder, if an excel file is missing the correct column it just doesnt load with no error so it can be difficult to troubleshoot when the data doesnt appear in the model. 

Is there best practise when setting up a workflow were you are recieving excel data from multiple team members to ensure the data will not cause errors or omissions?

[edhans · Accepted solution]

If the missing columns are needed for the report, nothing you can do about that as the data is missing and the report won't work.

However, if these are optional columns that you aren't using, then absolutely! Suggestions:

1. Remove the automatic "Changed Type" step that Power Query puts in. I actually disable that in global options now. What happens is PQ will apply data types to every column, some of which you don't care about and remove. But by explicitly converting the column data type, it will cause an error if that column is missing on the next refresh.
2. Select only the columns you need, and select Remove Other Columns from the Home Ribbon. Now missing columns will not be a factor as you never actually use them. By using Remove Other, you are actually telling Power Query to "Keep these columns" so you are not naming the unneeded columns in a "Remove this column" step.
3. Now you can set your data types manually for the columns you need to keep.

There are other ways to help make your report more robust. DataChant did a series of Pitfalls in Power Query few years ago and it is still 100% relevant. You can read those, or post back here with some more specific issues you are having.

[AnkitKukreja]

Hi @osabhaois 

I can feel that. What I practise is, I used to sent back those files initially and ask them to make the changes and keep the format as it was earlier, though it takes time but that's the best practise. 
So ideally they should keep the column name and they might leave it blank if there is no data but the column sequence and names should be similar as they were in last file.

Yeah it woun't show you an explanation for the error, but it would just say load failed or something, but if you'll check the same in Power Query you can find the root cause for the same.

Hope it helps.

Thanks

Ankit