Hi @rmbelda,
The gap between 35ms server-side and 3416ms client-side points to client infrastructure, not Eventhouse. Here are the likely causes worth investigating.
ServicePointManager connection limit
.NET defaults to 2 concurrent connections per endpoint in console applications. With 1000 workers hitting the same Eventhouse URL, you serialize through 2 sockets. Set ServicePointManager.DefaultConnectionLimit to 1024 at application startup, before any HTTP traffic begins. On .NET 8 with SocketsHttpHandler, also check MaxConnectionsPerServer on the underlying handler the Kusto SDK uses.
MSAL token acquisition serialization
A single service principal means a single token cache. MSAL serializes token requests through a per-authority semaphore. Under 1000-way concurrency, token acquisition queues even with cached tokens. Enable MSAL logging to confirm whether AcquireTokenForClient hits the in-memory cache or the network.
Token provider duplication across clients
Your 32 KustoConnectionStringBuilder instances likely instantiate separate token providers. On expiry, you get 32 parallel refresh attempts against AAD, with retry storms if AAD throttles. Share one TokenCredential or one ConfidentialClientApplication across all clients.
HttpClient pool fragmentation
The Kusto SDK creates HTTP infrastructure per client instance. 32 instances generate 32 connection pools, each governed by its own limit. Reduce to 1 to 4 ICslQueryProvider instances and tune the per-instance connection ceiling.
SNAT and ephemeral port pressure
1000 concurrent outbound connections from a single VM exhaust the ephemeral port range under TIME_WAIT. Run netstat -ano during the test and look for thousands of connections in TIME_WAIT to the Eventhouse endpoint. Enable HTTP keep-alive and verify reuse.
Gateway-side per-principal concurrency
The Eventhouse frontend applies per-principal concurrent request limits before queries reach the engine. EventhouseQueryLogs captures execution time only, so requests queued at the gateway are invisible in your 35ms measurement. Split the workload across two service principals as a test. If throughput roughly doubles, the bottleneck is per-principal at the gateway.
Diagnostic steps
- Capture client-side DiagnosticListener traces for HttpClient. Measure DNS resolution, TCP connect, TLS handshake, and request transmission separately to isolate where the 3381ms lives.
- Re-run with ServicePointManager.DefaultConnectionLimit set to 1024 and a single shared TokenCredential.
- Reduce SDK clients from 32 to 4 and compare throughput.
- Test with two service principals splitting load 500/500.
Run .show queries during the test and reconcile submitted client_request_id values against executed counts.
Theoretical throughput with 35ms execution and 1000 workers approaches 28,500 QPS. You observe 270 QPS, so something serializes roughly 100x of your concurrency. The two highest-probability culprits in my experience are the default connection limit and shared token cache contention.
I hope this will help to resolve your issue.
Thank you!
Proud to be a Super User!
📩 Need more help?
✔️ Don’t forget to Accept as Solution if this guidance worked for you.
💛 Your Like motivates me to keep helping
