Forum Discussion
Data getting too large - breaching 4 G memory usage. Data Framework wrong maybe?
Any other ideas? Is it that power bi for desktop just isn't a valid tool for large data sets?
Power BI is definately a memory hog. I use it with a 16Gb machine. 8gb works for most of our users. Restart if you get problems and make sure your memory has been release which can take a while. Close other memory hogs like browsers.
Obviously removed unneeded columns.
Aggregate in M to a lower grain before you load if you don't need all the detail.
Aim for a star schema if possible with id keys joins. Try adding a date dimension with your reporting month and join to your dates.
Turn off the Time Intelligence Auto Date/Time option if you don't need it as create date dimension in the background for every date field from the smallest to largest dates.
Also reduce the precision of times if you dont need them to be highly accurate.
Ideally load date and time seperatly both as integers e.g. mins after midnight.
I read here that ordering the data can help the compression if the data is very large. > 1million rows.
https://powerpivotpro.com/2016/02/sort-data-load-improved-compression/
here is some details of looking at what is using storage in your model.
https://www.sqlbi.com/articles/data-model-size-with-vertipaq-analyzer/
If you still too big then you may need to use either SSAS tabular on a server or try Azure Analysis Services (which can import your powerbi model)
https://azure.microsoft.com/en-gb/services/analysis-services/
However 800MB+ csv should be fine with the desktop version.
How big is the final powerbi doc? I have a couple of docs that are 0.5Gb.
Phil
- Lenihan8 years agoHelper III
the final size of the pbix file is only 127 mb. Our machines only have 8 GB of ram though.
Thank you for the information. I will look at what I need to do and what I can apply from your comments