Forum Discussion
benitovbreugel
Advocate II
11 months agoIssue spark resource usage is low & inefficient
Hi, We are running on an f8 Fabric capacity, with our own created spark environment. In our pipeline we are running the same notebook multiple times and it always uses the created environment. ...
tayloramy
Super User
11 months agoHi benitovbreugel,
Here’s what “low Spark efficiency + long waiting time” usually means in Fabric: your notebooks are waiting on session startup or the capacity job queue long before any cells execute. This is common when (a) custom environments are used (image build/attach time), (b) the capacity is at its Spark session/CPU limits so jobs get queued, or (c) you’re launching many separate sessions instead of reusing one via High Concurrency.
- Confirm if you’re queued or just slow to start. Open the run and check the Spark details page to see “Queued/Waiting” vs “Running” time. Use the Monitoring Hub > Spark application detail to pinpoint where the 15 minutes is spent (doc, overview).
- Minimize custom environment cost. Starter pools normally attach in ~5-10 seconds; custom environments (extra wheels) can push startup to minutes. Keep your custom env lean or try a test run on the starter pool to compare (doc, community example of slow starts with custom env: thread).
- Reuse one session. Turn on High Concurrency for notebooks/pipelines and keep all steps under the same user/workspace/environment so they pack into a single running session (concept, pipelines setup, notebooks setup).
- Reduce “many notebooks” > “one session”. Chain work in one notebook (use %run to call helpers) instead of multiple pipeline steps that each trigger their own session.
- Check capacity pressure. If the F8 is busy, Spark jobs queue even if your CU math says you “should fit.” Verify job queueing and throttling behavior (Spark job queueing, throttling/bursting). If you see persistent queue waits, consider scaling to F16 or staggering schedules.
- If it’s region/tenant specific, open a ticket. If starter pool also takes many minutes or you see unusual queue delays, contact Microsoft Support with run IDs and timestamps.
If you found this helpful, consider giving some Kudos. If I answered your question or solved your problem, mark this post as the solution.