How does Fabric handle concurrent real-time ML inference requests?

Hi vkeda29​,

I have not seen published hard numbers for concurrency or request size on Fabric ML model endpoints (they are still a preview feature, so limits can change), so I would treat this as something to measure rather than assume. A few points that shape the design though:

1. Multi-row requests are supported. The request body takes a list of input rows, so sending 10 records in one call is fine and is usually much cheaper than 10 separate calls. The model's endpoint page shows a sample request in the exact schema your MLflow signature expects; copy that rather than hand-building the payload.

2. Endpoint compute runs on your Fabric capacity. Inference consumes CUs from the same capacity as your other items. That is the real answer to "will it affect the existing application": if reports, Spark jobs and the endpoint share a capacity, a burst of inference traffic competes with them under the same throttling rules. For a production app I would put the model workspace on its own capacity.

3. Watch cold starts. Endpoints can auto-sleep when idle; the first call after sleep is slower. Either disable auto-sleep for production or send a lightweight warm-up call before peak periods.

4. Load test it yourself, ramping concurrency and logging latency and status codes, while watching the Capacity Metrics app at the same time:

import asyncio, time, httpx

URL = "<endpoint scoring url>"
HEADERS = {"Authorization": "Bearer <entra token>", "Content-Type": "application/json"}
BODY = {"inputs": [ {...} ] * 10}   # use the sample schema from the endpoint page

async def call(client, sem, out):
    async with sem:
        t = time.perf_counter()
        r = await client.post(URL, headers=HEADERS, json=BODY, timeout=60)
        out.append((r.status_code, time.perf_counter() - t))

async def run(concurrency, total=200):
    sem, out = asyncio.Semaphore(concurrency), []
    async with httpx.AsyncClient() as c:
        await asyncio.gather(*(call(c, sem, out) for _ in range(total)))
    lat = sorted(x[1] for x in out)
    errors = sum(1 for s, _ in out if s != 200)
    print(concurrency, "p50", round(lat[len(lat)//2], 3), "p95", round(lat[int(len(lat)*.95)], 3), "errors", errors)

for n in (1, 5, 10, 25, 50):
    asyncio.run(run(n))

If p95 latency or 429s climb before you reach your expected concurrency, the usual options are: batch more rows per call, add a small queue in your app, move to a larger or dedicated capacity, or, if you need explicit autoscale rules and SLAs, host the same MLflow model on an Azure ML managed online endpoint and keep Fabric for training and batch scoring (PREDICT in a notebook).

So my overall approach would be:

Batch rows per request (use the endpoint sample schema)
-> Dedicated capacity for the model workspace
-> Disable auto-sleep / warm up before peaks
-> Load test 1 -> 50 concurrent, watch p95, 429s and CU usage
-> If limits hit: queue + batch in app, bigger capacity, or Azure ML online endpoint

AI-assisted drafting: AI was used to help structure and phrase this response. I reviewed and validated the technical content before posting.

Hi vkeda29​,

The reply above is directionally right, but Microsoft now documents a few concrete scaling details that help with capacity planning.

Fabric ML model endpoints automatically scale based on incoming traffic, currently up to three compute nodes per active endpoint.

Microsoft documents that each active node consumes 5 CU-seconds per second, so one endpoint at full scale-out can consume 15 CU-seconds per second.

You can see that in the ML model endpoint documentation.

Bulk scoring is also supported. The scoring REST API accepts an inputs array, so sending 10 records in one request is a supported pattern.

For concurrent callers, Fabric manages the endpoint infrastructure and scales resource usage automatically, but Microsoft does not currently publish a universal maximum-concurrency or requests-per-second figure for these endpoints.

The scoring API does explicitly document HTTP 429 responses when the service limit is exceeded, together with a Retry-After header, so I would make sure the calling application implements bounded retry/backoff logic.

One other production point is Auto sleep.

It is enabled by default, and an idle endpoint can scale down after five minutes without incoming traffic. The first request that wakes it has additional latency, so Microsoft suggests turning Auto sleep off where consistent production response time is important.

For your XGBoost/LightGBM scenario, I would load-test the actual endpoint with representative traffic and measure:

• p50 / p95 / p99 latency
• throughput
• 429 responses
• single-row versus small-batch requests
• Fabric CU consumption while concurrency increases

Also monitor the Capacity Metrics app, where model endpoint usage appears under Model Endpoint.

So the practical model is:

managed autoscaling up to 3 nodes
→ higher concurrency can increase CU usage
→ no published universal RPS/concurrency guarantee
→ handle 429/Retry-After
→ disable Auto sleep if cold-start latency is unacceptable
→ size production from real load testing, not assumptions

AI-assisted drafting: AI was used to help structure and phrase this response. I reviewed and validated the technical content before posting.

Hi vkeda29​ ,

Could you check the suggestion provided above and let us know if you have any additional queries?

Thanks!!