topic Re: Notebook shows 'In Progress' in Monitor App even after it has returned with an error? in Data Engineering

Notebook shows 'In Progress' in Monitor App even after it has returned with an error?

IntegrateGuru — Thu, 12 Dec 2024 22:01:57 GMT

I've noticed that if I run a pyspark notebook with just the following code:

myVal = 1/0

And execute the notebook, the notebook will return very quickly with

ZeroDivisionError: division by zero

However, when I go into the monitor app to view the status of the notebook, the status is 'In Progress'.
It will remain 'In Progress' until the spark session times out, using our compute resources the entire time. Looking at the spark UI, it looks like right after the first executor is deallocated, a 2nd executor is allocated and then never deallocated, unexplicably.

What would cause this, and how can I avoid it?

The notebook should enter the error state, and immediately stop processing. It will not do so unless I manually cancel the notebook. Additionally, any subsequent runs of the notebook after the first is running, are not recorded by the Monitor App at all, while the original run is still going.

Re: Notebook shows 'In Progress' in Monitor App even after it has returned with an error?

IntegrateGuru — Thu, 12 Dec 2024 22:32:39 GMT

Seems like I've stumbled upon some interesting... features... of the pyspark notebooks.

If you exit unexpectedly from an interactive session - it will keep the session open until it times out, or you manually stop it.

This also effectively means that interactive notebooks can never enter a failed status AFAIK.

Re: Notebook shows 'In Progress' in Monitor App even after it has returned with an error?

Srisakthi — Fri, 13 Dec 2024 07:09:29 GMT

Hi @IntegrateGuru ,

The default timeout fro spark session is 20 mins, but you can configure this in workspace setting. We have to manually stop the spark session otherwise it will be available till the default timeout.

you can add mssparkutils.session.stop() to stop the session at the end of all your notebook .

Regards,

Srisakthi

Re: Notebook shows 'In Progress' in Monitor App even after it has returned with an error?

Anonymous — Fri, 13 Dec 2024 07:39:08 GMT

Hi @IntegrateGuru

Each notebook activity in the Monitor hub represents the spark application for a session. When you run a notebook, it initializes a spark session and creates a spark application. The "Status" represents the status of the session, rather than the running status of a code snippet in the notebook.

You can also use notebookutils.session.stop() to stop an interactive session and release resources occupied by the session. NotebookUtils (former MSSparkUtils) for Fabric - Microsoft Fabric | Microsoft Learn

Best Regards,
Jing

Re: Notebook shows 'In Progress' in Monitor App even after it has returned with an error?

nalinash — Fri, 13 Dec 2024 09:31:47 GMT

Hi @IntegrateGuru ,

Why you need to keep the session open when exit unexpectedly or having exception errors like division by zero or why you need to manually stop? You can automate this process after implementing either of these like exception handling or update the environment settings.

# Handling errors properly in the Notebook by catching errors explicitly and stop the Spark Session when exception occurs. It is terminating the session and free the resources immediately.

from pyspark.sql import SparkSession try: myVal = 1 / 0 except ZeroDivisionError as e: print(f"Error encountered: {e}") spark = SparkSession.getActiveSession() if spark: spark.stop()

# Check the notebook environment settings for timeout and error-handling configurations to terminate the Spark session on task failure. Inside the settings you can find out spark.dynamicAllocation.enabled and spark.dynamicAllocation.executeIdleTimeout. Also you can consider to reduce the executor idle timeout to free resources immediately when you have done.

Best regards,

Ash

Re: Notebook shows 'In Progress' in Monitor App even after it has returned with an error?

IntegrateGuru — Fri, 13 Dec 2024 14:04:14 GMT

Thanks, this seems correct.

Is there any way to set the status of a notebook to 'failed' then? We want to be able to view when our scripts run into errors, using an activator that listens to the activity status to go to a failed state.

Re: Notebook shows 'In Progress' in Monitor App even after it has returned with an error?

Anonymous — Mon, 23 Dec 2024 08:27:37 GMT

Hi @IntegrateGuru

If you want to view the contextual monitoring logs after running an interactive notebook, there are logs below every code cell. Notebook contextual monitoring and debugging - Microsoft Fabric | Microsoft Learn

If you want to set the status of a notebook to "failed", perhaps you can consider creating a data pipeline to invoke the notebook. Monitor the status of the notebook activity or the output of the notebook. One idea is:

At the ending cell of a notebook, use notebookutils.notebook.exit() to exit a notebook with an exit value like "Failed".
Create a data pipeline to invoke the notebook (Notebook activity). When you orchestrate a notebook in a pipeline that calls an exit() function, the notebook activity returns with an exit value, completes the pipeline run, and stops the Spark session.
Check the snapshot of a notebook activity in the data pipeline to know its status and snapshot in a run: Snapshot of Notebook from Pipeline in ... - Microsoft Fabric Community
(Alternative) You can also add a Fail activity after the notebook activity, and customize its error messages and error codes.

Hope this would be helpful.

Best Regards,
Jing
If this post helps, please Accept it as Solution to help other members find it. Appreciate your Kudos!