How does the Fabric handle the real-time ML inference requests?

Hi vkeda29​,

As we haven’t heard back from you, we wanted to kindly follow up to check if your issue is resolved? or let us know if you need any further assistance.

Thanks and regards,
Anjan Kumar Chippa

Hi vkeda29,

Thanks for raising this important question. Microsoft Fabric provides managed infrastructure for real-time ML inference, and there are several useful considerations for your XGBoost/LightGBM use case.

Here are a few points worth considering:

**1. Concurrent Requests and Auto Scaling**

Fabric automatically manages the underlying infrastructure and can scale an active ML endpoint up to three compute nodes based on incoming traffic.

**2. Bulk Predictions**

Yes, bulk scoring is supported. The scoring API accepts multiple input records through an `inputs` array. Sending 10 records in a single request is a supported approach and can help reduce individual request overhead.

**3. Request Limits and Throttling**

Microsoft does not currently document a universal maximum concurrency or guaranteed requests-per-second figure for these endpoints. The scoring API can return HTTP 429 when rate limits are exceeded. Applications should handle the `Retry-After` header and implement appropriate retry/backoff logic.

**4. Capacity Consumption**

Endpoint scaling directly affects Fabric Capacity Unit consumption. According to Microsoft's documentation, each active node consumes 5 CU-seconds per second, with a maximum of three nodes per endpoint.

**5. Auto Sleep and Response Time**

Auto sleep is enabled by default. After five minutes without incoming traffic, an endpoint can scale down to an idle state. The first request after inactivity may experience additional latency. For applications requiring consistent response times, consider disabling Auto sleep.

**6. Production Load Testing**

Before deploying to production, I recommend testing:

- Single-record versus batch predictions.

- Multiple simultaneous users.

- Average and 95th/99th percentile response times.

- Throughput and HTTP 429 errors.

- Fabric capacity consumption using the Capacity Metrics app.

**Microsoft Documentation:**

- [Serve Real-Time Predictions with ML Model Endpoints](https://learn.microsoft.com/en-us/fabric/data-science/model-endpoints)

- [Score ML Model Endpoint REST API](https://learn.microsoft.com/en-us/rest/api/fabric/mlmodel/endpoint/score-ml-model-endpoint)

- [Fabric REST API Throttling](https://learn.microsoft.com/en-us/rest/api/fabric/articles/throttling)

One important consideration: Fabric ML Model endpoints are currently in Preview, so I would recommend validating their performance against your application's expected traffic and SLA requirements before relying on them for critical production workloads.

**My question:** Have you considered testing batch predictions with 10–50 records alongside multiple concurrent users? Comparing both approaches could help identify the most efficient configuration.

Hope this helps!

Best regards,

Binita Fulpagare

Hi vkeda29​,

We wanted to kindly follow up to check if the solution I have provided for the issue worked? or let us know if you need any further assistance.

Thanks and regards,
Anjan Kumar Chippa

Hi vkeda29​,

As we haven’t heard back from you, we wanted to kindly follow up to check if the solution I have provided for the issue worked? or let us know if you need any further assistance.

Thanks and regards,
Anjan Kumar Chippa

Hi vkeda29​,

v-achippa​'s answer is correct that Fabric manages the underlying endpoint infrastructure; and there are a few documented details that are useful for sizing this.

Fabric ML model endpoints automatically scale based on incoming traffic, currently up to three compute nodes per active endpoint. Microsoft documents the scaling and capacity behaviour in the ML model endpoint guidance.

Bulk scoring in one request is also supported. The scoring REST API accepts an inputs array, so sending something like 10 records in one request is a supported pattern.

For concurrent callers, Fabric manages the serving infrastructure and scales the endpoint as traffic increases. However, I do not see Microsoft publishing a fixed requests-per-second or maximum-concurrency figure for these endpoints today.

The API does document HTTP 429 responses when the service rate limit is exceeded, together with a Retry-After header, so I would make sure the calling application implements retry/backoff handling.

There are also two production considerations I would pay attention to:

• Auto sleep is on by default. After five minutes without traffic, the endpoint can scale down, and the first request that wakes it has additional latency. Microsoft specifically suggests disabling Auto sleep when consistent production response time is required.
• Endpoint scaling consumes Fabric capacity. One active node currently consumes 5 CU-seconds per second, and an endpoint can scale up to three nodes under higher traffic.

For your XGBoost/LightGBM use case, I would therefore load-test the actual endpoint with representative payloads and concurrency rather than depend on a generic concurrency number. Measure at least p50/p95/p99 latency, 429 responses, throughput and CU usage in the Capacity Metrics app.

I would also test both single-record calls and small batches such as 10 records per request. Batching can reduce request overhead, but the best batch size will depend on your model and latency target.

One final point: ML model endpoints are still Preview, so for a production-critical application I would validate the behaviour and limits carefully before making them part of a strict application SLA.

AI-assisted drafting: AI was used to help structure and phrase this response. I reviewed and validated the technical content before posting.

Hi vkeda29​,

Thank you for reaching out to Microsoft Fabric Community.

Microsoft Fabric supports real-time inference through ML model endpoints. Activate an endpoint for the required model version and call its REST API to get predictions. Fabric manages the hosting infrastructure and scales the endpoint based on traffic. Disable auto-sleep if consistent response time is required

Since this feature is still in Preview, validate their performance and limitations before using them for a production-critical API. For workloads that require strict SLAs, use Azure Machine Learning managed online endpoint.

For reference, follow the below documentation:
Serve real-time predictions with ML model endpoints (Preview) - Microsoft Fabric | Microsoft Learn

Thanks and regards,
Anjan Kumar Chippa