Notebook shows 'In Progress' in Monitor App even after it has returned with an error?

Hi IntegrateGuru ,

Why you need to keep the session open when exit unexpectedly or having exception errors like division by zero or why you need to manually stop? You can automate this process after implementing either of these like exception handling or update the environment settings.

# Handling errors properly in the Notebook by catching errors explicitly and stop the Spark Session when exception occurs. It is terminating the session and free the resources immediately. 

from pyspark.sql import SparkSession

try:
    myVal = 1 / 0
except ZeroDivisionError as e:
    print(f"Error encountered: {e}")
    spark = SparkSession.getActiveSession()
    if spark:
        spark.stop()

# Check the notebook environment settings for timeout and error-handling configurations to terminate the Spark session on task failure. Inside the settings you can find out spark.dynamicAllocation.enabled and spark.dynamicAllocation.executeIdleTimeout. Also you can consider to reduce the executor idle timeout to free resources immediately when you have done.

Best regards,

Ash