Scalability options for Power BI on-prem Gateway - any options? Premium Per User env, refresh issue

[EngChi]

Are there any options available to help with spike load on the power bi gateway on-prem installations? anything available as a service? I have data warehouse processing running daily after which I invoke the refresh commands through Power BI APIs. These refreshes frequently /routinely fail (I think due to concurrent load on the gateway service/VM).

My entitlements are Premium Per User (not Premium capacity), rerunning the same calls to refresh datasets work if I space it out over time. I can scale up the server however I really need it spiking in capacity for about 15 minutes a day and rest of time it is not used at all.. 

Thank you

[v-cgao-msft]

Hi @EngChi ,

In addition to the above suggestions, there are several other areas you can start working on. For example, connection mode, refreshing at idle, optimizing queries, configuring incremental refreshing, and so on. For details, you can read the Best Practices section of the documentation.

Data refresh in Power BI - Power BI | Microsoft Docs

All these operations will make the refresh go in a more reliable direction, hope it helps.

Best Regards,
Gao
Community Support Team

If there is any post helps, then please consider Accept it as the solution to help the other members find it more quickly. If I misunderstand your needs or you still have problems on it, please feel free to let us know. Thanks a lot!

How to get your questions answered quickly -- How to provide sample data

[lbendlin]

"These refreshes frequently /routinely fail (I think due to concurrent load on the gateway service/VM)"

Run a powershell script on these gateway VMs to capture CPU load and available memory , and push the data into a streaming dataset.  That will give you a real-time-ish idea of the health of your gateway clusters.

- Use multiple VMs. At least two, better four or more

- make them big.  32GB RAM, 8 or more processor cores , fast disks   (probably something that you can dial up or down according to your load heatmap)

Also check the network performance.  Ideally the gateway VMs should live in proximity to the data sources.