Forum Discussion

NenadV's avatar
NenadV
Helper III
6 years ago

Filtering csv files from Azure Blob Storage

I have one question that might sound stupid, but I could not find answer anywere, so here it is. When we connect to data source, is whole data downloaded immediately when we click on transform button...
Need Help
NenadV's avatar
NenadV
Helper III
6 years ago

Yeah, over 300 mb just for listing the files

edhans's avatar
edhans
Community Champion
6 years ago

Then that is a function of Azure Blob Storage. I am very surprised to hear that.

 

I use Azure DataLake for CSV/Excel files and I can list hundreds of files in a few seconds in the Power Query editor. I suggest you consider moving the files to that storage medium. Even SharePoint doesn't download the files until you click Combine or click on a binary itself.

 

I use Blob Storage for images, but that is accessed via HTTPS URLs to images from the report. Power Query and Power BI's data model don't actually bring those images in. The service simply shows the images once the report loads.

  • NenadV's avatar
    NenadV
    Helper III
    6 years ago

    This data is generated on daily basis so it amount will increase quickly. I will consider moving it somewhere else for sure.

    • edhans's avatar
      edhans
      Community Champion
      6 years ago

      See this article published just yesterday on using Azure Data Lake gen 2. Seems perfect for your scenario. I know ADL does not download file contents until it begins the combine operation.