Microsoft fabrics concurrent execution of notebooks

[c_avu]

I have 3 notebooks and running these three notebooks concurrently using mssparkutils.notebook.runMultiple(DAG)

all of 3 notebooks returning error message using exit() function

--> mssparkutils.notebook.exit(error_message)

my dag notebook:

from notebookutils import mssparkutils

DAG = {

"activities": [

{

"name": "extract_customers",

"path": "extract_customers",

"timeoutPerCellInSeconds": 120,

"args": {"rows": 1000},

},

{

"name": "extract_products",

"path": "extract_products",

"timeoutPerCellInSeconds": 120,

"args": {"rows": 5000},

},

{

"name": "products_table",

"path": "products_table",

"timeoutPerCellInSeconds": 90,

"retry": 1,

"retryIntervalInSeconds": 10,

"dependencies": ["extract_products"]

}

],

"timeoutInSeconds": 1800,

"concurrency": 5

}

mssparkutils.notebook.runMultiple(DAG, {"displayDAGViaGraphviz": True})

i'm running this dag notebook in databricks, Any idea how to catch exist values returned by internal notebooks in datafactory pipeline. 

Usecase is need to send an email if there are any exit values present

any idea how to do this ?

[Anonymous · Accepted solution]

Hi, @c_avu 

I have found some relevant documentation that I hope will help you resolve this issue:

1.Firstly, regarding referencing notebook exit values as variables in your data pipeline, you can refer to a similar issue here:

Referencing notebook exit value as a variable in a... - Microsoft Fabric Community

Below is a screenshot from the relevant documentation:

2.Secondly, for information on how to send emails using Azure Data Factory or Azure Synapse pipelines, you can refer to the official documentation here:

How to send email - Azure Data Factory & Azure Synapse | Microsoft Learn

3.Lastly, you might need to understand how to transform data by running Databricks notebooks:

Transform data with Databricks Notebook - Azure Data Factory & Azure Synapse | Microsoft Learn

If you find a solution that works, you are welcome to share it with us, so that members of our community who have similar questions as you can find that answer more quickly.

Thank you in advance for your efforts.

Best Regards,

Leroy Lu

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.