Forum Discussion

IAMCS's avatar
IAMCS
Icon for Helper I rankHelper I
11 months ago
Solved

notebook failed before execution started because Fabric could not create a Spark (Livy) session.

I have 2 pipelines , which simply run notebooks in my medallion architecture and refresh the semantic model. they used to run fine until yesterday, below is the error: Notebook execution...
workspaces
  • wardy912's avatar
    wardy912
    11 months ago

    Sounds like another job was running at the same time using the available CUs. Check the capacity metrics report. Glad to hear it's working now.

wardy912's avatar
wardy912
Icon for Super User rankSuper User
11 months ago

Hi IAMCS 

 

 Has the capacity been changed at all? If the SKU has been reduced or if there are several processes running at the same time, it can cause this error. It can also happen if the workspace has been moved to a new capacity.

 Try creating a new spark environment from the dropdown at the top of the notebook

 

This will ensure the environment is in the correct capacity.

In the setup, go to compute to give yourself more spark driver cores if you want to improve speed of the notebook.

 

 Make sure to check your capacity metrics app, see if it's not just a standard capacity usage issue.

Also, check with colleagues, they may be running an intensive process that you're not aware of.

 

--------------------------------

I hope this helps, please give kudos and mark as solved if it does!

 

Connect with me on LinkedIn.

Subscribe to my YouTube channel for Fabric/Power Platform related content!