Importing folder takes very long

[haykp]

Dear Community,

Please help to understand why importing folder takes so long and how I can speed up.

I have some folder hierarchy which contains neede csv files. The csv files are located below 3 levels of hierarchy. So I import the top folder hierarchy than in Transform gui 

a) filter all .csv files ( takes really long to filter)

b) filder all the files which start with the given name e.g. fileName_ (again this operation takes very long)

c) Press combine to combine all these file

 Here the powerbi got stack. 

It becoms extremly slow. 

Please can someone help to debug and speed up.

I make another experiment - copied all the csv files in the seperate folder and imported that folder. In such case powerbi works very fast and normal.

Thanks

Hayk

[Alex_Sawdo]

You could write a script that would combine each of the files into one, thus saving Power Query some processing power for doing all this for you. When pulling data, it's best to try and do all of your data collection/transformations as close to the source as possible and this is an excellent example as to why. If this isn't an option, you'd probably have to stick with multiple queries to break up the processing for a possible performance increase. 

[Thejeswar]

@haykp ,

One Option that I could think of is, Instead of selecting the folder at the top most hierarchy, You can create multiple connections ponting to the specific folders from where you want the csv files imported.

Once that is done, you can append all these queries together. This will make everrything a single query for you to report on

[haykp]

Thanks for the answer.

Can you please explain more about your suggestion?

Do you mean creating multiple Queryies?

[Thejeswar]

Yes, You need to create multiple queries each referring to one folder from where you are getting the file. After doing the below steps, append the queries

a) filter all .csv files 

b) filter all the files which start with the given name e.g. fileName_ 

c) Append all the queries to make them into one

[haykp]

Thanks for the answer.

Acutally I am creating the same way, just each of the query takes ages. Also the a),b) steps takes too long.

[Thejeswar]

@haykp 

When you split the folder path and pull the data using different queries, in a ideal case, the first 2 steps time taken should get decreased to very little. Even in that case it takes a lot of time, It should be that there are huge number of files in the location that you are trying to access. 

If you don't have huge number of files in that location, I don't think it is a problem with Power BI connector. Something else is causing this issue