Forum Discussion

Mark3333333's avatar
Mark3333333
New Member
4 months ago
Solved

Notebook does not stop in monitor

Dear community,

 

i have a question: i've started a notebook. So far so good. And in visual studio it ends after 1 min.

But in my monitor the notebook it is still marked as "In progress".

And i cannot start another job. What is going on?

 

Has somebody an idea?

best regards
Mark

notebook
spark job definition
  • Mark3333333's avatar
    Mark3333333
    3 months ago

    Hi,

    thanks for your answers. With 

    notebookutils.session.stop() i can work.
    best regards

12 Replies

  • Dev_Dholakia's avatar
    Dev_Dholakia
    Icon for Resolver III rankResolver III
    4 months ago

    Hi Mark3333333 ,

     

    If I understood correctly, your code finished on the client side but the Monitor hub still shows the run as “In progress.”


    That’s because Monitor tracks the Spark session, not the cell. The session stays alive until it times out or is stopped, which is why new jobs are blocked.


    A few things to try:

    • Open Monitor hub → your notebook run → View details / Spark UI and check if the Spark application is still active.
    • From the notebook, stop the session explicitly with notebookutils.session.stop() (or use the Stop session button in the session dropdown).
    • If the run is stuck in Monitor, use Cancel on the activity row.
    • Wait for the session timeout (default 20 min) or lower it in Workspace settings → Data Engineering → Spark → Session timeout.
    • Wrap your code in try/except and call spark.stop() on failure so the session releases when an exception is thrown.
    • If it stays stuck after all that, raise a support ticket with workspace name, notebook name, run/application ID, timestamp, and capacity.

    Quickcheck: are you running this as an interactive Fabric notebook, from VS Code (Fabric extension), or via a pipeline activity? Behavior differs slightly.


    Useful references:

    Monitoring hub
    NotebookUtils session management
    Develop, execute, and manage notebooks

    Found this useful? A quick Kudos goes a long way.

    Got what you needed? Marking this as the Accepted Solution helps others land on the right answer faster when they search for the same thing!

    • Mark3333333's avatar
      Mark3333333
      New Member
      4 months ago

      HI,

      i start the script interactive in visual studio.

       

      br

      Mark

    • Mark3333333's avatar
      Mark3333333
      New Member
      4 months ago

      Hi, when i start via pipeline its ok. When i start it via visual studio or fabric manually it does not stop.

      Whats the reason?

      • Dev_Dholakia's avatar
        Dev_Dholakia
        Icon for Resolver III rankResolver III
        4 months ago

        Hey Mark3333333 , thanks for the extra detail, that’s usually expected.

        Interactive runs (VS Code or manual in Fabric) keep the Spark session alive after the code finishes so you can reuse it, while pipeline runs release the session automatically. Monitor tracks the Spark session/app, not just the cell run — so it stays “In progress” until the session ends, which can block other jobs.

        A Few things yoi can try:

        • End the session in code: notebookutils.session.stop()
        • Or use the Stop session button in the notebook toolbar.
        • Open the Monitor row → Spark UI to check if the app is actually still running or just idle.
        • Check the interactive session timeout setting.

        Are you launching from the Fabric Data Engineering extension in VS Code, or from a local .py/.ipynb?
        and does adding notebookutils.session.stop() at the end change what you see in Monitor?

  • v-kathullac's avatar
    v-kathullac
    Icon for Community Support rankCommunity Support
    3 months ago

    Thanks IrtizaSM , Dev_Dholakia   for Addressing the issue.

     

    Hi  Mark3333333  ,
    we would like to follow up to see if the solution provided by the super user resolved your issue. Please let us know if you need any further assistance.

     

    Thanks,

    Chaithanya.

    • IrtizaSM's avatar
      IrtizaSM
      Frequent Visitor
      4 months ago

      Hi Mark3333333 ,

      Since the notebook is already completing successfully, I'd also verify whether the Spark application has already finished. If it has, but the Monitor still reports In progress, that usually suggests the monitoring state is out of sync rather than the notebook still running.

      Could you check if that's the case in your environment?

  • v-kathullac's avatar
    v-kathullac
    Icon for Community Support rankCommunity Support
    3 months ago

    Thanks @IrtizaSM , @Dev_Dholakia   for Addressing the issue.

     

    Hi  @Mark3333333  ,
    we would like to follow up to see if the solution provided by the super user resolved your issue. Please let us know if you need any further assistance.

     

    Thanks,

    Chaithanya.

  • Mark3333333's avatar
    Mark3333333
    New Member
    3 months ago

    Hi,

    thanks for your answers. With 

    notebookutils.session.stop() i can work.
    best regards
  • v-kathullac's avatar
    v-kathullac
    Icon for Community Support rankCommunity Support
    3 months ago

    Hi Mark3333333 ,

     

    Thankyou for the update that your issue got resolved Let us know if you need any further assistance.

     

    Thanks,

    Chaithanya.