Forum Discussion
Filtering csv files from Azure Blob Storage
I am filtering files by their name, before combine files step. I tried many combinations, just to test how it is working. When I do not filter anything and combine files and load everything, power bi download over 300 mb. When I do not filter anything and do not combine files (I just load info about files, like name, type, date created...) power bi again download over 300 mb, like in previous case. Even when I remove binary column. So i am not sure that power bi do not download content before combine files step. And when I filter files before combine files step, and load data, power bi download about 30mb (I leave just 10-15 files). So it seems to me that filtering of files by name is done in Azure blob storage and that whole content of files is downloaded when you load data, regardless of presence of combine files step. I do not have much experince with this, so I am not sure who all of that is done.
You are saying that when you go to the storage blob and get a listing of the files, Power BI is downloading 300MB just to show you the folder listing?
Yeah, over 300 mb just for listing the files
Then that is a function of Azure Blob Storage. I am very surprised to hear that.
I use Azure DataLake for CSV/Excel files and I can list hundreds of files in a few seconds in the Power Query editor. I suggest you consider moving the files to that storage medium. Even SharePoint doesn't download the files until you click Combine or click on a binary itself.
I use Blob Storage for images, but that is accessed via HTTPS URLs to images from the report. Power Query and Power BI's data model don't actually bring those images in. The service simply shows the images once the report loads.
This data is generated on daily basis so it amount will increase quickly. I will consider moving it somewhere else for sure.