mssparkutils.notebook.runMultiple doesn't return notebook Status if notebook fails

[anawast]

I would like to run multiple notebooks in parallel, and collect the status of each of the notebook runs. While using mssparkutils.notebook.runMultiple, as long as all notebooks succeed there are no issues. 

However, the failure of any one notebook results in the entire command failing and thus the inability to collect the result. 

Is there any way via which I can use mssparkutils.notebook.runMultiple to ensure that i collect the status of each notebook, even if there are failures that exist among the notebooks I am running in parallel? 

P.S: I would like to avoid putting all commands in the child notebook in try catch blocks and running notebook.exit commands on any exception. 

[anawast · Accepted solution]

These don't run in parallel. But i guess I can multithread for that. Essentially use something like what is documented here: https://learn.microsoft.com/en-us/azure/databricks/_extras/notebooks/source/parallel-notebooks.html

[v-huijiey-msft]

Hi @anawast ,

Is this a workaround?

Can you accept your own answer as a solution? This would be a great help to people with similar problems.

Thank you for your cooperation.

Best Regards,
Yang
Community Support Team

If there is any post helps, then please consider Accept it as the solution  to help the other members find it more quickly.
If I misunderstand your needs or you still have problems on it, please feel free to let us know. Thanks a lot!

[v-huijiey-msft]

Hi @anawast ,

It is possible to run each subnotebook in the main notebook using mssparkutils.notebook.run and capture the state of each subnotebook in the main notebook.

The code is as follows:

def run_notebook(notebook_path):
    try:
        mssparkutils.notebook.run(notebook_path, 3600)
        return {"notebook": notebook_path, "status": "Success"}
    except Exception as e:
        return {"notebook": notebook_path, "status": "Failed", "error": str(e)}

notebooks = ["Notebook 2", "Notebook 8"]
results = [run_notebook(nb) for nb in notebooks]

# result
for result in results:
    print(result)

Here is a screenshot of my successful run:

I've modified the code for notebook 8 a bit, here's a screenshot of one of the notebooks running and reporting an error, click on notebook 2 to see the result of a successful run.

If you have any other questions please feel free to contact me.

Best Regards,
Yang
Community Support Team

If there is any post helps, then please consider Accept it as the solution  to help the other members find it more quickly.
If I misunderstand your needs or you still have problems on it, please feel free to let us know. Thanks a lot!

[jjaeger94]

This will not run concurrently will it? 

The benefit of run multiple is that it runs the notebooks in parallel