topic Re: Trigger Alerts Notification for successful/failed Pipelines in Pipelines

Trigger Alerts Notification for successful/failed Pipelines

Anonymous — Wed, 19 Mar 2025 07:15:05 GMT

Hi Community!

I have a scenario where I have a master pipeline which has multiple child pipelines within it.

For Example:

Pipeline A

Child pipeline(Pipeline A) - Pipeline B

Child pipeline(Pipeline B) - Pipeline C

Pipeline A has some activities and is then connected to a child pipeline Pipeline B, Pipeline B has few activities with another child pipeline Pipeline C. Now these pipelines are interconnected and have dependencies. I want to ensure that I get an email notifications triggered upon successful/ failed runs. I did find some solutions in the community but none of them suits my scenario. Also I have complicated pipelines, therefore I cannot work on below scenarios as well(includes too many activities/ doesn't suit my scenario):
https://www.c-sharpcorner.com/article/error-logging-and-the-art-of-avoiding-redundant-activities-in-msft-fabric-data-p/

https://community.fabric.microsoft.com/t5/Data-Pipeline/Monitor-pipeline-runs-with-alerts/m-p/4141013

I was thinking of using a single notebook activity and using Fabric API's to parameterize my pipelines and capture logs and use email notifications. However, I thought there might be better solutions.

Please let me know if there are any other ways.

Thanks in Advance!
Hrishi

Re: Trigger Alerts Notification for successful/failed Pipelines

Anonymous — Thu, 20 Mar 2025 01:34:53 GMT

Hi Anonymous

To address your scenario in Microsoft Fabric, where you have a master pipeline with multiple interconnected child pipelines and want to trigger email notifications upon success or failure, here are some potential solutions:

Using Fabric API with a Notebook Activity
Your idea of using a single notebook activity to parameterize pipelines and capture logs is a good approach. Here's how you can implement it:

Use the Fabric API to monitor pipeline runs programmatically.
Create a notebook activity in your pipeline that:
Fetches the pipeline run status using the Fabric API.
Logs the status (success/failure) for each pipeline and its child pipelines.
Sends an email notification using an SMTP library (e.g., Python's smtplib) or a third-party email service like SendGrid.
Parameterize the notebook to handle different pipelines dynamically.

Using Webhooks for Notifications
You can use webhooks to trigger notifications based on pipeline events.

Configure a webhook in Microsoft Fabric to listen for pipeline run events (success/failure).
Use a service like Azure Logic Apps, Power Automate, or a custom webhook handler to process the event and send email notifications.
Ensure the webhook is triggered for both the master and child pipelines.

Custom Logging and Notification Pipeline
Create a separate pipeline dedicated to logging and notifications.

At the end of each pipeline (master and child), add an activity to log the run status to a centralized location (e.g., Azure Blob Storage, SQL Database).
Trigger the logging and notification pipeline after the master pipeline completes.
The logging pipeline reads the run statuses and sends email notifications.

Using Azure Monitor Alerts
Leverage Azure Monitor to track pipeline runs and send alerts.

Enable diagnostic settings for your pipelines to send logs to Azure Monitor.
Create an alert rule in Azure Monitor to trigger on specific conditions (e.g., pipeline failure).
Configure the alert to send email notifications.

Combining Fabric API and Logic Apps
If you want a more robust solution, combine the Fabric API with Azure Logic Apps.

Use the Fabric API to fetch pipeline run statuses.
Trigger a Logic App workflow to process the statuses and send email notifications.
Schedule the Logic App to run periodically or trigger it from the pipeline.

Regards,

Nono Chen

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

Re: Trigger Alerts Notification for successful/failed Pipelines

Anonymous — Fri, 21 Mar 2025 05:02:16 GMT

Hi Anonymous ,

Thank you for the logic, I happen to use notebook at pipeline run time by using a wait activity(as the status might take some time to reflect since we are using API). I happen to use the notebook during pipeline run where I fetch my latest run. I am able to extract this. However, I am trying to use Outlook activity to push the notification. The issue is I am not sure how to convert my string output to Table.

This is the structure of my notebook df output:

Main Pipeline Name	Master Pipeline Name	Pipeline Name	Invoke Pipeline Name	Pipeline ID	Status	Error Message
PL_Parameter_	Main Pipeline	Invoke pipeline1	PL_Parameter_	xyz-abc	Failed	{Error_Message_will_be_displayed}
PL_Parameter_	PL_Parameter_	Invoke pipeline1	PL_Test_1	efg-fhi	Failed	{Error_Message_will_be_displayed}

I am using mssparkutils to capture this notebook output(exitValue) as a string and pass it to set variable(optional) then to Outlook activity. But since it is string, I am not sure how I can convert this string to table.

Also, if we want to use SMTP, can I use FabricRestClient instead of passing username and password.
Can you please share a sample structure for SMTP code as I don't want to pass username and password.

Thanks in advance!
Hrishi

Re: Trigger Alerts Notification for successful/failed Pipelines

Anonymous — Fri, 21 Mar 2025 05:08:37 GMT

Also, we are only looking to integrate this as a complete Fabric solution (we don't want to use Azure).

Re: Trigger Alerts Notification for successful/failed Pipelines

Anonymous — Wed, 02 Apr 2025 11:46:44 GMT

cant we use power automate for this ?

Re: Trigger Alerts Notification for successful/failed Pipelines

v-saisrao-msft — Thu, 08 May 2025 11:36:15 GMT

Hi Anonymous

Thanks for the detailed context you're on the right track with using a notebook and Fabric APIs. To convert your DataFrame output into a format suitable for the Outlook activity, I recommend using df.to_html(index=False) within your notebook to transform the DataFrame into an HTML table string. You can then pass this HTML string using mssparkutils.notebook.exit(), capture it in a Set Variable activity, and use it as the email body in your Outlook activity (assuming it supports HTML content). This approach maintains formatting and ensures readability in email notifications. Regarding SMTP, FabricRestClient currently doesn’t support credential-free SMTP directly. If avoiding username/password is critical, consider using the Outlook activity (if supported in your tenant) or triggering a webhook to Power Automate for notifications, which provides secure and flexible email delivery without handling credentials directly.

If this post helps, then please give us ‘Kudos’ and consider Accept it as a solution to help the other members find it more quickly.

Thank you.

Re: Trigger Alerts Notification for successful/failed Pipelines

v-saisrao-msft — Tue, 13 May 2025 04:58:10 GMT

Hi Anonymous ,
I wanted to check if you had the opportunity to review the information provided. Please feel free to contact us if you have any further questions. If my response has addressed your query, please accept it as a solution and give a 'Kudos' so other members can easily find it.
Thank you.

Re: Trigger Alerts Notification for successful/failed Pipelines

v-saisrao-msft — Fri, 16 May 2025 06:02:38 GMT

Hi Anonymous,

May I ask if you have resolved this issue? If so, please mark the helpful reply and accept it as the solution. This will be helpful for other community members who have similar problems to solve it faster.

Thank you.

Re: Trigger Alerts Notification for successful/failed Pipelines

v-saisrao-msft — Mon, 19 May 2025 09:50:21 GMT

Hi Anonymous ,

We haven’t heard back from you regarding your issue. If it has been resolved, please mark the helpful response as the solution and give a ‘Kudos’ to assist others. If you still need support, let us know.

Thank you.