Forum Discussion

IAMCS's avatar
IAMCS
Icon for Helper I rankHelper I
11 months ago
Solved

notebook failed before execution started because Fabric could not create a Spark (Livy) session.

I have 2 pipelines , which simply run notebooks in my medallion architecture and refresh the semantic model. they used to run fine until yesterday, below is the error: Notebook execution...
workspaces
  • wardy912's avatar
    wardy912
    11 months ago

    Sounds like another job was running at the same time using the available CUs. Check the capacity metrics report. Glad to hear it's working now.

shashiPaul1570_'s avatar
shashiPaul1570_
Icon for Responsive Resident rankResponsive Resident
11 months ago

Hi IAMCS ,

The failure is happening because the notebook is requesting more Spark resources than the pool supports.
Your job requested 48 cores / 336 GB RAM, while the selected pool only allows 16 cores / 112 GB RAM, so Fabric cannot create a Livy session and the pipeline stops before execution.

 How to fix?

  • Reduce Spark session settings (Run → Configure session): lower driver size, executor size, or executor count.

  • OR use a larger Spark pool if the workload requires more compute.

  • Also confirm the pipeline activity isn’t overriding the notebook’s Spark settings.

If this explanation helps, please consider giving a kudos 👍 and marking it as the Accepted Solution ✅ so it can help others as well.

Regards,
Shashi Paul