How does Fabric handle concurrent real-time ML inference requests?

Hi vkeda29​,

v-saisrao-msft​'s answer covers the important points well.

One additional sizing detail from Microsoft’s current documentation is that an active Fabric ML endpoint can automatically scale up to three compute nodes based on incoming traffic.

Each active node currently consumes 5 CU-seconds per second, so a single endpoint at full scale-out can consume 15 CU-seconds per second.

Microsoft documents that behavior in the ML model endpoint guidance.

That is useful for capacity planning, but I would still avoid assuming a fixed concurrency or requests-per-second limit because Microsoft does not currently publish a universal value for those.

For production testing, I would measure:

• p50/p95/p99 response time;
• throughput;
• 429 responses and Retry-After behavior;
• single-row versus small-batch requests;
• Fabric CU consumption while concurrency increases.

Also keep Auto sleep in mind. It is enabled by default, and an idle endpoint can scale down after five minutes, which adds latency to the first request that wakes it. Microsoft specifically suggests disabling Auto sleep when consistent response time is important.

AI-assisted drafting: AI was used to help structure and phrase this response. I reviewed and validated the technical content before posting.