Notebook shows 'In Progress' in Monitor App even after it has returned with an error?

[IntegrateGuru]

I've noticed that if I run a pyspark notebook with just the following code: 

myVal = 1/0

And execute the notebook, the notebook will return very quickly with

ZeroDivisionError: division by zero

 
However, when I go into the monitor app to view the status of the notebook, the status is 'In Progress'. 
It will remain 'In Progress' until the spark session times out, using our compute resources the entire time. Looking at the spark UI, it looks like right after the first executor is deallocated, a 2nd executor is allocated and then never deallocated, unexplicably.


What would cause this, and how can I avoid it?

The notebook should enter the error state, and immediately stop processing. It will not do so unless I manually cancel the notebook. Additionally, any subsequent runs of the notebook after the first is running, are not recorded by the Monitor App at all, while the original run is still going. 

[nalinash]

Hi @IntegrateGuru ,

Why you need to keep the session open when exit unexpectedly or having exception errors like division by zero or why you need to manually stop? You can automate this process after implementing either of these like exception handling or update the environment settings.

# Handling errors properly in the Notebook by catching errors explicitly and stop the Spark Session when exception occurs. It is terminating the session and free the resources immediately. 

from pyspark.sql import SparkSession

try:
    myVal = 1 / 0
except ZeroDivisionError as e:
    print(f"Error encountered: {e}")
    spark = SparkSession.getActiveSession()
    if spark:
        spark.stop()

# Check the notebook environment settings for timeout and error-handling configurations to terminate the Spark session on task failure. Inside the settings you can find out spark.dynamicAllocation.enabled and spark.dynamicAllocation.executeIdleTimeout. Also you can consider to reduce the executor idle timeout to free resources immediately when you have done.

Best regards,

Ash

[v-jingzhan-msft]

Hi @IntegrateGuru 

Each notebook activity in the Monitor hub represents the spark application for a session. When you run a notebook, it initializes a spark session and creates a spark application. The "Status" represents the status of the session, rather than the running status of a code snippet in the notebook. 

You can also use notebookutils.session.stop() to stop an interactive session and release resources occupied by the session. NotebookUtils (former MSSparkUtils) for Fabric - Microsoft Fabric | Microsoft Learn

Best Regards,
Jing

[IntegrateGuru]

Thanks, this seems correct. 

Is there any way to set the status of a notebook to 'failed' then? We want to be able to view when our scripts run into errors, using an activator that listens to the activity status to go to a failed state.

[v-jingzhan-msft]

Hi @IntegrateGuru 

If you want to view the contextual monitoring logs after running an interactive notebook, there are logs below every code cell. Notebook contextual monitoring and debugging - Microsoft Fabric | Microsoft Learn 

If you want to set the status of a notebook to "failed", perhaps you can consider creating a data pipeline to invoke the notebook. Monitor the status of the notebook activity or the output of the notebook. One idea is:

1. At the ending cell of a notebook, use notebookutils.notebook.exit() to exit a notebook with an exit value like "Failed". 
2. Create a data pipeline to invoke the notebook (Notebook activity). When you orchestrate a notebook in a pipeline that calls an exit() function, the notebook activity returns with an exit value, completes the pipeline run, and stops the Spark session.
3. Check the snapshot of a notebook activity in the data pipeline to know its status and snapshot in a run: Snapshot of Notebook from Pipeline in ... - Microsoft Fabric Community
4. (Alternative) You can also add a Fail activity after the notebook activity, and customize its error messages and error codes. 

Hope this would be helpful. 

Best Regards,
Jing
If this post helps, please Accept it as Solution to help other members find it. Appreciate your Kudos!

[Srisakthi]

Hi @IntegrateGuru ,

The default timeout fro spark session is 20 mins, but you can configure this in workspace setting. We have to manually stop the spark session otherwise it will be available till the default timeout.

you can add mssparkutils.session.stop() to stop the session at the end of all your notebook .

Regards,

Srisakthi

[IntegrateGuru]

Seems like I've stumbled upon some interesting... features... of the pyspark notebooks. 

If you exit unexpectedly from an interactive session - it will keep the session open until it times out, or you manually stop it. 

This also effectively means that interactive notebooks can never enter a failed status AFAIK.