Forum Discussion
powerbi gateway - sizing
Anonymous,
I think the issue is with system specs. 2 vCPUs and 8GB of RAM is very, very low when dealing with large datasets. Instead of table row size, how much total data are you dealing with in terms of GBs?
We have clients with 50GB+ datasets, and we're using Azure Databricks with 15 vCPUs and 56GB of RAM.
i had a support ticket in with microsoft about this, but there solutions didnt work.
in the end i increased the memory on the server to 16gb and it works fine now - peaks at around 8.3gb
we are now using t2.xlarge.
Interesting observations about the gateway whilst debugging the problem.
i would say the process is split into 3
a) get the data via the odbc connector
b) mashup the data using the mashup container
c) send the data to powerbi service
In our case step b) is the memory intensive part and doesnt handle running out of memory very elegantly
Also I would recommend putting the gateway as near to the data source as possible, to avoid network issues as the volume of data received (step a) is far greater than the volume of data sent (step c)
I wasn't referring to the PBIX file - because of compression, there's not a direct ratio between the amount of data and the size of the file. I was referring to the amount of data you're pulling in from your datasource, in GB.
Databricks is not a Power BI service, it's a data processing platform. I was just using it as an example of the specs we deal with.
apologies.
The main table that we are processing is 9gb in redshift, and most of its columns are encoded, so i guess depending on how these things work, it could be higher by the time it gets to powerbi gateway.
To make the import more manageable i load it in 10 separate imports and then append these inside powerbi - not sure if this is a good idea or not.