Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

Register now to learn Fabric in free live sessions led by the best Microsoft experts. From Apr 16 to May 9, in English and Spanish.

Reply
Karl23
New Member

Unbearably slow loading of CSV files

I must be missing something here. 

 

I have a 16-core CPU, 128 gigs of RAM, and the SSD can read at 3 gigabytes per second.

 

I have a file that's about a gig in size. It loads in less than 1 second if I open it in a text editor (all of it loaded into RAM). It contains around 20 columns of text and numbers. 

 

Tableau loads it in about 8 seconds (~100 mb per second).

 

If I just open PowerBI, press get data, load the file (zero transformations at all) it loads it at about 3 mb/second, over 25x slower than tableau. There are no load errors and I have disabled background load and all of that which I read can speed up loads. As soon as you make a single change to anything in the data transformation step, have to load it all again which again takes 10+ minutes. It is simply unbearable to work with, and the data size is tiny! 1 gb! It should take SECONDS to load locally on a modern computer!

 

I don't understand. PowerBI's charting tools are so good. All of it is super great, and you can tell that the underlying calculation engine is super performant and written by experts. Once loaded, the data calculates in an instant. However, the fact that data load simply doesn't work makes power bi completely useless for anything other than hobby projects. What am I missing here, are you all suffering with this, having to leave the computer for hours at a time while the **bleep** thing works as if it was running on a computer from 1995?

 

I have another report that loads about 5 gigs of data from flat CSV files on the online service (with premium capacity) - it takes around TWO HOURS to refresh, and again, there are no joins, nothing except loading a flat CSV. Loading (and parsing it) with a custom c# program takes about 4 seconds on my machine. What the hell is wrong with PowerBI, it has literally everything going for it but then because data load simply doesn't work it is 100% unusable.

 

Every time there is a monthly update I am thinking "surely now they have adressed the fact that the entire tool literally doesn't work" but no, it's some useless update to some third party connector that 4 people use. What am I missing here??? 

 

 

3 REPLIES 3
PBIandME
Advocate I
Advocate I

@Karl23 
Im having similar issues but by refresshing the a dataflow connector that is connected to csv's, I think in this months update they definetely broke somthing because this was not an issue for the past year for me.

I have a support case with microsoft and will update you once they help, (usually they cant help and then it gets pushed to actual microsoft employees).

ppm1
Solution Sage
Solution Sage

In addition to reading in the data, Power BI is also compressing it into columnar storage during that time (check out the size of your pbix file). This compression enables the fast analysis once loaded. If your two hour refresh involves date-based files, you can leverage incremental refresh to speed things up. You may be able to adapt the technique shown in this video to do that.

https://www.youtube.com/watch?v=IVMdg16yBKE

 

Pat

Microsoft Employee

Come on Pat. This is CPU utilization while loading data. I could probably compress it quicker myself with pen and paper.

 

Karl23_0-1677938940455.png

 

Helpful resources

Announcements
Microsoft Fabric Learn Together

Microsoft Fabric Learn Together

Covering the world! 9:00-10:30 AM Sydney, 4:00-5:30 PM CET (Paris/Berlin), 7:00-8:30 PM Mexico City

PBI_APRIL_CAROUSEL1

Power BI Monthly Update - April 2024

Check out the April 2024 Power BI update to learn about new features.

April Fabric Community Update

Fabric Community Update - April 2024

Find out what's new and trending in the Fabric Community.