Forum Discussion

anawast's avatar
anawast
Microsoft Employee
2 years ago
Solved

mssparkutils.notebook.runMultiple doesn't return notebook Status if notebook fails

I would like to run multiple notebooks in parallel, and collect the status of each of the notebook runs. While using mssparkutils.notebook.runMultiple, as long as all notebooks succeed there are no issues. 

However, the failure of any one notebook results in the entire command failing and thus the inability to collect the result. 


Is there any way via which I can use mssparkutils.notebook.runMultiple to ensure that i collect the status of each notebook, even if there are failures that exist among the notebooks I am running in parallel? 

P.S: I would like to avoid putting all commands in the child notebook in try catch blocks and running notebook.exit commands on any exception. 

Data Integration

4 Replies

  • Anonymous's avatar
    Anonymous
    Not applicable
    2 years ago

    Hi anawast ,

     

    It is possible to run each subnotebook in the main notebook using mssparkutils.notebook.run and capture the state of each subnotebook in the main notebook.

     

    The code is as follows:

    def run_notebook(notebook_path):
    try:
        mssparkutils.notebook.run(notebook_path, 3600)
        return {"notebook": notebook_path, "status": "Success"}
    except Exception as e:
        return {"notebook": notebook_path, "status": "Failed", "error": str(e)}

notebooks = ["Notebook 2", "Notebook 8"]
results = [run_notebook(nb) for nb in notebooks]

# result
for result in results:
    print(result)

     

    Here is a screenshot of my successful run:

     

    I've modified the code for notebook 8 a bit, here's a screenshot of one of the notebooks running and reporting an error, click on notebook 2 to see the result of a successful run.

     

    If you have any other questions please feel free to contact me.

     

    Best Regards,
    Yang
    Community Support Team

     

    If there is any post helps, then please consider Accept it as the solution  to help the other members find it more quickly.
    If I misunderstand your needs or you still have problems on it, please feel free to let us know. Thanks a lot!

    • jjaeger94's avatar
      jjaeger94
      Helper II
      1 year ago

      This will not run concurrently will it? 

      The benefit of run multiple is that it runs the notebooks in parallel 

    • Anonymous's avatar
      Anonymous
      Not applicable
      2 years ago

      Hi anawast ,

       

      Is this a workaround?

       

      Can you accept your own answer as a solution? This would be a great help to people with similar problems.

       

      Thank you for your cooperation.

       

      Best Regards,
      Yang
      Community Support Team

       

      If there is any post helps, then please consider Accept it as the solution  to help the other members find it more quickly.
      If I misunderstand your needs or you still have problems on it, please feel free to let us know. Thanks a lot!