Forum Discussion

c_avu's avatar
c_avu
Frequent Visitor
1 year ago
Solved

Microsoft fabrics concurrent execution of notebooks

I have 3 notebooks and running these three notebooks concurrently using mssparkutils.notebook.runMultiple(DAG)

 

all of 3 notebooks returning error message using exit() function

--> mssparkutils.notebook.exit(error_message)

 

my dag notebook:

 

from notebookutils import mssparkutils
DAG = {
"activities": [
{
"name": "extract_customers",
"path": "extract_customers",
"timeoutPerCellInSeconds": 120,
"args": {"rows": 1000},
},
{
"name": "extract_products",
"path": "extract_products",
"timeoutPerCellInSeconds": 120,
"args": {"rows": 5000},
},
{
"name": "products_table",
"path": "products_table",
"timeoutPerCellInSeconds": 90,
"retry": 1,
"retryIntervalInSeconds": 10,
"dependencies": ["extract_products"]
}
],
"timeoutInSeconds": 1800,
"concurrency": 5
}
mssparkutils.notebook.runMultiple(DAG, {"displayDAGViaGraphviz": True})

i'm running this dag notebook in databricks, Any idea how to catch exist values returned by internal notebooks in datafactory pipeline. 

Usecase is need to send an email if there are any exit values present

any idea how to do this ?


Tips and Tricks

1 Reply