Forum Discussion

haykp's avatar
haykp
Icon for Helper I rankHelper I
3 years ago

Importing folder takes very long

Dear Community,

 

Please help to understand why importing folder takes so long and how I can speed up.

 

I have some folder hierarchy which contains neede csv files. The csv files are located below 3 levels of hierarchy. So I import the top folder hierarchy than in Transform gui 

a) filter all .csv files ( takes really long to filter)

b) filder all the files which start with the given name e.g. fileName_ (again this operation takes very long)

c) Press combine to combine all these file

 Here the powerbi got stack. 

It becoms extremly slow. 

 

Please can someone help to debug and speed up.

 

I make another experiment - copied all the csv files in the seperate folder and imported that folder. In such case powerbi works very fast and normal.

 

Thanks

Hayk

Need Help

6 Replies

  • Thejeswar's avatar
    Thejeswar
    Icon for Super User rankSuper User
    3 years ago

    haykp ,

    One Option that I could think of is, Instead of selecting the folder at the top most hierarchy, You can create multiple connections ponting to the specific folders from where you want the csv files imported.

     

    Once that is done, you can append all these queries together. This will make everrything a single query for you to report on

    • haykp's avatar
      haykp
      Icon for Helper I rankHelper I
      3 years ago

      Thanks for the answer.

      Can you please explain more about your suggestion?

      Do you mean creating multiple Queryies?

      • Thejeswar's avatar
        Thejeswar
        Icon for Super User rankSuper User
        3 years ago

        Yes, You need to create multiple queries each referring to one folder from where you are getting the file. After doing the below steps, append the queries

         

        a) filter all .csv files 

        b) filter all the files which start with the given name e.g. fileName_ 

        c) Append all the queries to make them into one

         

  • Alex_Sawdo's avatar
    Alex_Sawdo
    Icon for Resolver II rankResolver II
    3 years ago

    You could write a script that would combine each of the files into one, thus saving Power Query some processing power for doing all this for you. When pulling data, it's best to try and do all of your data collection/transformations as close to the source as possible and this is an excellent example as to why. If this isn't an option, you'd probably have to stick with multiple queries to break up the processing for a possible performance increase.