Gateway settings to maximize the remote server utilization to refresh datasets

[vg_pbiuser]

Hello Everyone,

I am trying to refresh the datasets(large datasets) on PBI service. The dataset is built on on-prem database datasource and refreshed using an on-prem gateway. When I logon to the remote server and look into the resource usage, I do not see much of the memory/cpu being used by the gateway processes. Does anyone got a chance to tweak the gateway settings to make the gateway processes behave differently. Please let me know if anyone got a chance to work on a similar issue.

Thanks!!

[Anonymous]

Hi @vg_pbiuser ,

If a gateway cluster with load balancing enabled receives a request from one of the cloud services (like Power BI), it randomly selects a gateway member. If this member is already at or over the throttling limit set for CPU or memory, another member within the cluster is selected. If all members within the cluster are in the same state, the request fails.

To enable this feature, a gateway admin should update the following settings in the Microsoft.PowerBI.DataMovement.Pipeline.GatewayCore.dll.config file available in the _Program Files\On-premises data gateway_ folder.

Please refer to the following document for more details.

https://docs.microsoft.com/en-us/data-integration/gateway/service-gateway-high-availability-clusters...

https://docs.microsoft.com/en-us/data-integration/gateway/service-gateway-performance-cpu

Best Regards

Community Support Team _ Polly

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly

[Anonymous]

Hi @vg_pbiuser ,

Can you tell us which process is using 100% of the CPU? If it's Microsoft.Mashup.Container*, then it's a similar problem to https://social.technet.microsoft.com/Forums/en-US/63159ce7-1252-49ea-891f-d01d7fd4e1be/microsoft-mas... Otherwise it's different.

The DisableSystemCounterReport is by default set to False to enable additional logging for memory and CPU system counters. This creates the System Counter Aggregation Report file. To turn this feature off, switch the flag to True.

Best Regards

Community Support Team _ Polly

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly

[vg_pbiuser]

The issue is with the gateway not using the resources completely.

[nitishsh91]

Hi @vg_pbiuser 

Please update 2 parameters in Microsoft.PowerBI.DataMovement.Pipeline.GatewayCore.dll.config :

CPUUtilizationPercentageThreshold

MemoryUtilizationPercentageThreshold

Please have a look at below url:

https://docs.microsoft.com/en-us/data-integration/gateway/service-gateway-performance

[vg_pbiuser]

Yes Nitish and thanks for the response. We did look into these parameters. However, by default it was disabled and which means it has to utilize 100 % of CPU or Memory. We tried to explicitly set the parameters to 80%, to notice any changes in the usage. We haven't seen any improvement in terms of usage. Is there anything else that could effect this ?