StreamBeforeRequestCompletes Option in On Premise Data Gateway not working

[Anonymous]

Hello,

we've set up the standard On-Premises data gateway (version 3000.222.5, May 2024) in an Azure VM (Standard B8als v2 (8 vcpus, 16 GiB memory)) to connect Power BI with Azure MySQL for flexible Server.

We've noticed that the disk write operations are quite intense and to improve gateway performance we tried enabling the `StreamBeforeRequestCompletes` option in the XML config and restarted the service. But no matter how often we restarted the VM or gateway in the meantime, the Gateway is still writing files in the `Spooler` directory of the application and disk operations haven't changed. Other settings like `MashupDefaultPoolContainerMaxCount` had effects after changing.

Is the `StreamBeforeRequestCompletes` option limited to specific data sources, or are we missing something else?

Happy to provide more information and thankful for any hint!

[lbendlin · Accepted solution]

StreamBeforeRequestCompletes  is only applicable when you are actually streaming (ie without transforms  or with query folding). All other queries still require disk activity for mashups etc.

What you should do is change the directory for spooler and logs to a different disk, so that it doesn't thrash your system disk.

[lbendlin · Accepted solution]

StreamBeforeRequestCompletes  is only applicable when you are actually streaming (ie without transforms  or with query folding). All other queries still require disk activity for mashups etc.

What you should do is change the directory for spooler and logs to a different disk, so that it doesn't thrash your system disk.

[Anonymous]

Thanks for the quick response!
We will double-check the setup of the queries to see if your topics apply.
The spooler folder is currently not filling up, the files appear and disappear after some seconds to minutes.

[Anonymous · Accepted solution]

Your ideas is so great  @lbendlin 
Hi,  @Anonymous 
I'm wondering if you have resolved your issue by now. If so, it would be great if you could mark the helpful replies as Answered. This way, other members who have similar questions can find the solutions more easily.
In addition, you can also adjust the performance of the gateway with the CPU of the server. We recommend that you read the following documents to understand the performance logs of your gateway and optimize the performance of your gateway based on them:
Adjust gateway performance based on server CPU | Microsoft Learn

Configure gateway disk space | Microsoft Learn

Monitor and optimize on-premises data gateway performance | Microsoft Learn

 

How to Get Your Question Answered Quickly

If it does not help, please provide more details with your desired output and pbix file without privacy information (or some sample data) .

Best Regards

Jianpeng Li

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.