Concurrent spark sessions

Hello Kar_c 

• F8 SKU provides:
• 16 Spark vCores (base).
• Burst multiplier: 3x → up to 48 Spark vCores.
• Each Medium Starter Pool session (default) uses 16 vCores (2 nodes × 8 vCores/node).
• Without bursting: Only 1 session can run (16 vCores / 16 vCores per session).
• With bursting: Up to 3 sessions (48 vCores / 16 vCores per session

 If capacity is exceeded:
• Interactive notebooks: Fail with “Too Many Requests For Capacity” errors.
• Scheduled jobs (via Pipelines/Scheduler): Queue up to 8 jobs

Each pipeline activity starts a separate session. For example:
• 2 pipelines running 2 notebooks each = 4 sessions (exceeds F8’s 3-session limit with bursting).

try implementing these for better concurrency 

Use High Concurrency Mode
• Allows up to 5 notebooks to share a single Spark session

Leverage `NotebookUtils.RunMultiple`
• Run multiple notebooks in parallel within a single session to avoid capacity limits

Single-node Spark Pool:
• Configure autoscale to 1 node → 8 vCores per session.
• F8 can run 6 concurrent sessions (48 vCores / 8 vCores per session)
Custom Small Pool:
• Half the size of Medium → 4 vCores per session.