Loading a folder and parallel

[arcegabriel]

I have a query that read ~6M lines from maybe 12 CSVs on a folder. I noticed it reads each csv sequentially. In Power Query if I "Refresh All" Queries it takes 1min 05 seconds. 

Is this normal speed? 

[arcegabriel]

For testing purposes, I combined all the CSVs into one big CSV. Loading is a little faster at 53seconds (12 seconds faster than pulling individual files). 

[v-yingjl]

Hi @arcegabriel ,

When loading a combined file into power query, it would be recogzied as a single data source and create a single query for it so the loading time for it is single.

When loading multiple files from the folder parallely, power query would create each single query for each file and the loading time would be little slower than a combined file.

In addition, as a data source, not only it has data in it but also has metadata like credentials, frames etc.. which would also affect the performance when loading into power query. A combined file only needs a single load while multilpe files need multiple load.

Best Regards,
Community Support Team _ Yingjie Li
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

[arcegabriel]

So if you have chosen to to do a query from a folder, it would be one query for all files and hence there never would be parallel loading.

If I have queried each file individually it would be true parallel

There is no way around "folder" query to read in parallel

[Fowmy]

@arcegabriel 

It depends on the number of transformations and the columns you are loading. However, 12 Million around 1minuit looks okay for me. You can also improve the speed by unticking the downloading  background  data

[arcegabriel]

That was a test where I just created a pbix that loaded the data. Nothing else (that query was "Enable to Load")

For clarity 6million lines is the result after reading 12 files (not each file)

[Fowmy]

I typed it incorrectly,