Forum Discussion

NenadV's avatar
NenadV
Helper III
6 years ago

Filtering csv files from Azure Blob Storage

I have one question that might sound stupid, but I could not find answer anywere, so here it is. When we connect to data source, is whole data downloaded immediately when we click on transform button...
Need Help
edhans's avatar
edhans
Community Champion
6 years ago

You are saying that when you go to the storage blob and get a listing of the files, Power BI is downloading 300MB just to show you the folder listing?

NenadV's avatar
NenadV
Helper III
6 years ago

Yeah, over 300 mb just for listing the files

  • edhans's avatar
    edhans
    Community Champion
    6 years ago

    Then that is a function of Azure Blob Storage. I am very surprised to hear that.

     

    I use Azure DataLake for CSV/Excel files and I can list hundreds of files in a few seconds in the Power Query editor. I suggest you consider moving the files to that storage medium. Even SharePoint doesn't download the files until you click Combine or click on a binary itself.

     

    I use Blob Storage for images, but that is accessed via HTTPS URLs to images from the report. Power Query and Power BI's data model don't actually bring those images in. The service simply shows the images once the report loads.

    • NenadV's avatar
      NenadV
      Helper III
      6 years ago

      This data is generated on daily basis so it amount will increase quickly. I will consider moving it somewhere else for sure.

      • edhans's avatar
        edhans
        Community Champion
        6 years ago

        See this article published just yesterday on using Azure Data Lake gen 2. Seems perfect for your scenario. I know ADL does not download file contents until it begins the combine operation.