Forum Discussion
How does Fabric handle concurrent real-time ML inference requests?
## Regarding the scalability and performance of Fabric ML model endpoints.
For our Use case, we're planning to deploy an XGBoost/LightGBM model in Fabric and consume the prediction endpoint through the application.
Can the API support bulk predictions. For example, we may send 10 records in a single request, but we could also have multiple users calling the same endpoint simultaneously.
I understand Fabric manages the underlying compute and supports automatic scaling, but I wanted to clarify a few things:
How does Fabric handle concurrent requests to the same ML endpoint? Is the load balancing managed automatically?
Are there any concurrency, throughput, or request-size limits we should consider?
How does the scaling behave when the number of concurrent requests increases, and what impact does this have on response time and capacity consumption?
Are there any recommended approaches for load testing or configuring the endpoint for production applications?
Our main concern is ensuring that the prediction service doesn't affect the existing performance of the application.
Has anyone tested Fabric ML endpoints with concurrent requests or used them for a similar application integration?
2 Replies
v-saisrao-msftCommunity Support
HI vkeda29,
Microsoft Fabric ML model endpoints support XGBoost and LightGBM model flavors, subject to the documented schema limitations. The scoring API accepts multiple input records in a single request. Fabric manages the endpoint infrastructure and scales resource usage based on incoming traffic, up to three nodes per endpoint; however, documented maximum concurrency and guaranteed throughput limits are not specified. Handle HTTP 429 responses using the Retry-After header, and consider disabling auto sleep to avoid wake-up delays after inactivity. Before production deployment, load-test representative batch sizes and concurrent requests, monitoring response times, errors, throughput, and Fabric capacity consumption.
Serve real-time predictions with ML model endpoints (Preview) - Microsoft Fabric | Microsoft Learn
Endpoint - Score ML Model Endpoint - REST API (MLModel) | Microsoft Learn
Throttling - Microsoft Fabric REST APIs | Microsoft Learn
What is the Microsoft Fabric Capacity Metrics app? - Microsoft Fabric | Microsoft LearnThank you.
- ShivekMaharaj
Community Champion
Hi vkeda29,
v-saisrao-msft's answer covers the important points well.
One additional sizing detail from Microsoft’s current documentation is that an active Fabric ML endpoint can automatically scale up to three compute nodes based on incoming traffic.
Each active node currently consumes 5 CU-seconds per second, so a single endpoint at full scale-out can consume 15 CU-seconds per second.
Microsoft documents that behavior in the ML model endpoint guidance.
That is useful for capacity planning, but I would still avoid assuming a fixed concurrency or requests-per-second limit because Microsoft does not currently publish a universal value for those.
For production testing, I would measure:
- p50/p95/p99 response time;
- throughput;
- 429 responses and Retry-After behavior;
- single-row versus small-batch requests;
- Fabric CU consumption while concurrency increases.
Also keep Auto sleep in mind. It is enabled by default, and an idle endpoint can scale down after five minutes, which adds latency to the first request that wakes it. Microsoft specifically suggests disabling Auto sleep when consistent response time is important.AI-assisted drafting: AI was used to help structure and phrase this response. I reviewed and validated the technical content before posting.
