How does the Fabric handle the real-time ML inference requests?

Hi vkeda29​,

Thank you for reaching out to Microsoft Fabric Community.

Microsoft Fabric supports real-time inference through ML model endpoints. Activate an endpoint for the required model version and call its REST API to get predictions. Fabric manages the hosting infrastructure and scales the endpoint based on traffic. Disable auto-sleep if consistent response time is required

Since this feature is still in Preview, validate their performance and limitations before using them for a production-critical API. For workloads that require strict SLAs, use Azure Machine Learning managed online endpoint.

For reference, follow the below documentation:
Serve real-time predictions with ML model endpoints (Preview) - Microsoft Fabric | Microsoft Learn

Thanks and regards,
Anjan Kumar Chippa

Hi vkeda29​,

v-achippa​'s answer is correct that Fabric manages the underlying endpoint infrastructure; and there are a few documented details that are useful for sizing this.

Fabric ML model endpoints automatically scale based on incoming traffic, currently up to three compute nodes per active endpoint. Microsoft documents the scaling and capacity behaviour in the ML model endpoint guidance.

Bulk scoring in one request is also supported. The scoring REST API accepts an inputs array, so sending something like 10 records in one request is a supported pattern.

For concurrent callers, Fabric manages the serving infrastructure and scales the endpoint as traffic increases. However, I do not see Microsoft publishing a fixed requests-per-second or maximum-concurrency figure for these endpoints today.

The API does document HTTP 429 responses when the service rate limit is exceeded, together with a Retry-After header, so I would make sure the calling application implements retry/backoff handling.

There are also two production considerations I would pay attention to:

• Auto sleep is on by default. After five minutes without traffic, the endpoint can scale down, and the first request that wakes it has additional latency. Microsoft specifically suggests disabling Auto sleep when consistent production response time is required.
• Endpoint scaling consumes Fabric capacity. One active node currently consumes 5 CU-seconds per second, and an endpoint can scale up to three nodes under higher traffic.

For your XGBoost/LightGBM use case, I would therefore load-test the actual endpoint with representative payloads and concurrency rather than depend on a generic concurrency number. Measure at least p50/p95/p99 latency, 429 responses, throughput and CU usage in the Capacity Metrics app.

I would also test both single-record calls and small batches such as 10 records per request. Batching can reduce request overhead, but the best batch size will depend on your model and latency target.

One final point: ML model endpoints are still Preview, so for a production-critical application I would validate the behaviour and limits carefully before making them part of a strict application SLA.

AI-assisted drafting: AI was used to help structure and phrase this response. I reviewed and validated the technical content before posting.

Hi vkeda29​,

As we haven’t heard back from you, we wanted to kindly follow up to check if the solution I have provided for the issue worked? or let us know if you need any further assistance.

Thanks and regards,
Anjan Kumar Chippa