Hi everyone,
I am trying to use the custom prompt AI function using Fabric's default LLM settings. I am trying to send over 100k records through the LLM but this is resulting in immediate capacity issues due to the volume of records being sent. I am curious if anyone has also experienced these issues and if so, how did you get around them? Does using my own key and endpoint set up in Azure AI foundary work or will that also create capacity issues. More details below:
Environment
Workspace SKU: F128 (autoscale for Spark ON)
Feature: Custom Prompt AI (Fabric’s default LLM endpoint)
Use case: Classifying ~100 K customer-address records in one pass
Data size: ≈ 1 M tokens per run (each record ~10 tokens × 100 K)
Problem
Whenever I call the endpoint with the full batch, capacity utilization spikes > 300 % and Fabric immediately throttles the workspace. Even if I chunk the dataframe into 5–10 calls, the bursts still blow the CU budget and the overage burndown takes hours.
What I’ve tried
Tactic Result
| ai.generate_response() on the full Spark DF | Instant overage / throttling |
| Micro-batching (10 K rows) | 8–10 bursts, still > 200 % CU |
| Switching to a lower-temp model | Token cost drops, CU spike unchanged |
| Using Autoscale for Spark pools | Helps after the fact (burndown), not at call time |
Questions for the community
Batch / stream patterns – Has anyone found a sweet-spot batch size or sliding-window pattern that lets you keep CU < 100 % while feeding large dataframes to the LLM?
Async vs. sync calls – Can I fire off smaller async requests and aggregate the responses later without paying the “concurrent CU penalty” in one big spike?
Queue or orchestrator – Does Fabric offer a built-in queue for LLM calls (similar to Synapse Spark job queue) or is everyone rolling their own (e.g., Delta table + Logic Apps / Data Factory orchestrator)?
Model / capacity separation – Is it possible to point Custom Prompt AI to a pay-as-you-go Azure OpenAI resource so the heavy LLM work lands outside the Fabric capacity?
Throttle-aware retry logic – Any sample code that backs off when Peak % > X and resumes when burndown < Y?
I’d love to hear how others are handling high-volume LLM inference in Fabric without upgrading to an even larger SKU or waiting hours for overages to clear.
