Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

Enhance your career with this limited time 50% discount on Fabric and Power BI exams. Ends August 31st. Request your voucher.

Reply
c_avu
Frequent Visitor

Microsoft fabrics concurrent execution of notebooks

I have 3 notebooks and running these three notebooks concurrently using mssparkutils.notebook.runMultiple(DAG)

 

all of 3 notebooks returning error message using exit() function

--> mssparkutils.notebook.exit(error_message)

 

my dag notebook:

 

from notebookutils import mssparkutils
DAG = {
"activities": [
{
"name": "extract_customers",
"path": "extract_customers",
"timeoutPerCellInSeconds": 120,
"args": {"rows": 1000},
},
{
"name": "extract_products",
"path": "extract_products",
"timeoutPerCellInSeconds": 120,
"args": {"rows": 5000},
},
{
"name": "products_table",
"path": "products_table",
"timeoutPerCellInSeconds": 90,
"retry": 1,
"retryIntervalInSeconds": 10,
"dependencies": ["extract_products"]
}
],
"timeoutInSeconds": 1800,
"concurrency": 5
}
mssparkutils.notebook.runMultiple(DAG, {"displayDAGViaGraphviz": True})

i'm running this dag notebook in databricks, Any idea how to catch exist values returned by internal notebooks in datafactory pipeline. 

Usecase is need to send an email if there are any exit values present

any idea how to do this ?


1 ACCEPTED SOLUTION
Anonymous
Not applicable

Hi, @c_avu 

 

I have found some relevant documentation that I hope will help you resolve this issue:

 

1.Firstly, regarding referencing notebook exit values as variables in your data pipeline, you can refer to a similar issue here:

Referencing notebook exit value as a variable in a... - Microsoft Fabric Community

Below is a screenshot from the relevant documentation:

vlinyulumsft_0-1726812583756.png

2.Secondly, for information on how to send emails using Azure Data Factory or Azure Synapse pipelines, you can refer to the official documentation here:

How to send email - Azure Data Factory & Azure Synapse | Microsoft Learn

 

3.Lastly, you might need to understand how to transform data by running Databricks notebooks:

Transform data with Databricks Notebook - Azure Data Factory & Azure Synapse | Microsoft Learn

 

If you find a solution that works, you are welcome to share it with us, so that members of our community who have similar questions as you can find that answer more quickly.

 

Thank you in advance for your efforts.

 

Best Regards,

Leroy Lu

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

View solution in original post

1 REPLY 1
Anonymous
Not applicable

Hi, @c_avu 

 

I have found some relevant documentation that I hope will help you resolve this issue:

 

1.Firstly, regarding referencing notebook exit values as variables in your data pipeline, you can refer to a similar issue here:

Referencing notebook exit value as a variable in a... - Microsoft Fabric Community

Below is a screenshot from the relevant documentation:

vlinyulumsft_0-1726812583756.png

2.Secondly, for information on how to send emails using Azure Data Factory or Azure Synapse pipelines, you can refer to the official documentation here:

How to send email - Azure Data Factory & Azure Synapse | Microsoft Learn

 

3.Lastly, you might need to understand how to transform data by running Databricks notebooks:

Transform data with Databricks Notebook - Azure Data Factory & Azure Synapse | Microsoft Learn

 

If you find a solution that works, you are welcome to share it with us, so that members of our community who have similar questions as you can find that answer more quickly.

 

Thank you in advance for your efforts.

 

Best Regards,

Leroy Lu

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

Helpful resources

Announcements
July 2025 community update carousel

Fabric Community Update - July 2025

Find out what's new and trending in the Fabric community.

July PBI25 Carousel

Power BI Monthly Update - July 2025

Check out the July 2025 Power BI update to learn about new features.