Forum Discussion
Filtering csv files from Azure Blob Storage
All filtering is done locally. Azure DataLake and Storage Blob do nothing on their end.
You can filter the file names though before you begin the COMBINE operation. No file contents are downloaded until you click combine.
So in this article, you could filter the files before you combine them assuming your can filter by file date or name. If you need to filter based on data in the files then you have no choice but to filter after the combine.
As a rule Power Query only shows the first 1,000 records to aid in speeding up development, but with CSV files, it often gets bogged down. I've seen it take 4-5min per step as I am working with them.
I am filtering files by their name, before combine files step. I tried many combinations, just to test how it is working. When I do not filter anything and combine files and load everything, power bi download over 300 mb. When I do not filter anything and do not combine files (I just load info about files, like name, type, date created...) power bi again download over 300 mb, like in previous case. Even when I remove binary column. So i am not sure that power bi do not download content before combine files step. And when I filter files before combine files step, and load data, power bi download about 30mb (I leave just 10-15 files). So it seems to me that filtering of files by name is done in Azure blob storage and that whole content of files is downloaded when you load data, regardless of presence of combine files step. I do not have much experince with this, so I am not sure who all of that is done.
