Unable to run PySpark Notebook because of apparent Spark Setting issues

Hi,

Based on the error message, this is not actually a notebook or Livy issue.
It is a Spark pool resource validation error.

The error clearly indicates that the Spark session is requesting more compute resources than what your selected Spark pool allows.

In your case:

• Claimed cores: 80

• Claimed memory: 560 GB

• Pool limit: 24 cores / 168 GB

So Fabric is rejecting the session creation before execution starts.

This usually happens when:

• Executor count × executor cores exceeds pool capacity

• Driver + executor memory total exceeds pool memory limit

• Environment or notebook level settings are not aligned with pool node size

• Dynamic allocation or previous session settings override notebook config

In Microsoft Fabric, notebook-level Spark configuration must always stay within the boundaries of the selected pool node size and pool scaling limits. Even if you configure executors in the notebook, Fabric will validate them against pool limits during session creation.

According to Microsoft documentation, session-level compute settings can only be configured within the limits of the selected Spark pool node size and memory boundaries:
https://learn.microsoft.com/en-us/fabric/data-engineering/environment-manage-compute

Also, Spark pool node sizes define the available vCores and memory per node, and workloads must fit inside those limits:
https://learn.microsoft.com/en-us/fabric/data-engineering/create-custom-spark-pools

Additionally, the %%configure command must be executed at the beginning of the notebook (before session starts), otherwise settings may not apply:
https://learn.microsoft.com/en-us/fabric/data-engineering/author-execute-notebook

What I would check step by step

1️⃣ Validate Pool Configuration

• Node size (Medium / Large / etc.)

• Min / Max nodes

• Autoscale settings

• Capacity SKU limits

2️⃣ Validate Notebook Configuration
Make sure total requested resources are within pool capacity:

Example:

Total cores = numExecutors × executorCores + driverCores
Total memory = numExecutors × executorMemory + driverMemory

3️⃣ Check Environment vs Notebook Conflicts
If you have Environment-level Spark settings, they might override notebook config.

4️⃣ Restart Spark Session
After changing config, restart session or rerun notebook from first cell.

5️⃣ Consider Dynamic Allocation
If workloads vary, enabling dynamic allocation may help optimize executor usage within pool limits.

Example Safe Configuration (Medium Node Example)

If pool = Medium (8 vCores / ~56 GB per node):

%%configure -f
{
  "executorMemory": "28g",
  "executorCores": 4,
  "driverMemory": "28g",
  "driverCores": 4,
  "numExecutors": 2
}

Key Insight

Fabric Spark works like this:

Infrastructure Layer → Pool (Nodes / Capacity / SKU)
Execution Layer → Executors / Driver (must fit inside pool)

Environment or Notebook configs cannot override pool infrastructure limits.

If this suddenly started happening (even though notebook worked before), I would also check:

• Pool configuration changes by admin

• Capacity SKU changes

• Environment default compute changes

• New autoscale limits

• Concurrent running notebooks consuming pool resources

Hope this helps.
Let me know if you can share your pool configuration and capacity SKU — then I can help calculate the safe executor configuration range.

Best regards