Forum Discussion
Power Query folder combine -- file combining -- faster?
- 5 years ago
peterhui50 can you not use this:
=List.ContainsAny({"A".."Z"}, Text.ToList([Column1]))It returns true or false. It is case sensitive. So the "xxxxx" value returns false since it isn't in A-Z
Changing it to this returns Yes or No.
if List.ContainsAny({"A".."Z"}, Text.ToList([Column1])) then "Yes" else "No"And changing it to this is case insensitive.
if List.ContainsAny({"A".."Z", "a".."z"}, Text.ToList([Column1])) then "Yes" else "No" - 5 years ago
peterhui50 - post back with performance enhancements. I don't know how much more efficient a List.Contains with Text.ToList is vs a single if/then/else with Text.Contains, but it surely must be more efficient than 26 of them!
Thanks to mahoneypat for asking the right question. I was just thinking originally of a lot of data. Looking at the transformations should have been at the top of my list as well.
I've not done any benchmarking, but to me it seems CSV is faster than XLSX when accessing large amounts of data.
You will have to test and experiment, but I will say this:
- Remove unnecessary columns early in the process (or from the source is even better)
- Filter early in the process if anything is filtered out so later transformation happen on the smaller recordset.
- Consider Azure DataLake for file storage. It seems to me to work faster and more reliably than files in SharePoint with large datasets.