Trigger Alerts Notification for successful/failed Pipelines

Question

Hi Community!&nbsp;I have a scenario where I have a master pipeline which has multiple child pipelines within it.&nbsp;For Example:Pipeline AChild pipeline(Pipeline A) - Pipeline BChild pipeline(Pipeline B) - Pipeline C&nbsp;Pipeline A has some activities and is then connected to a child pipeline Pipeline B, Pipeline B has few activities with another child pipeline Pipeline C. Now these pipelines are interconnected and have dependencies. I want to ensure that I get an email notifications triggered upon successful/ failed runs. I did find some solutions in the community but none of them suits my scenario. Also I have complicated pipelines, therefore I cannot work on below scenarios as well(includes too many activities/ doesn't suit my scenario):https://www.c-sharpcorner.com/article/error-logging-and-the-art-of-avoiding-redundant-activities-in-msft-fabric-data-p/https://community.fabric.microsoft.com/t5/Data-Pipeline/Monitor-pipeline-runs-with-alerts/m-p/4141013&nbsp;&nbsp;I was thinking of using a single notebook activity and using Fabric API's to parameterize my pipelines and capture logs and use email notifications. However, I thought there might be better solutions.&nbsp;Please let me know if there are any other ways.Thanks in Advance!Hrishi

anonymous · Answer

Hi&nbsp;Anonymous&nbsp;
&nbsp;
To address your scenario in Microsoft Fabric, where you have a master pipeline with multiple interconnected child pipelines and want to trigger email notifications upon success or failure, here are some potential solutions:
&nbsp;
Using Fabric API with a Notebook ActivityYour idea of using a single notebook activity to parameterize pipelines and capture logs is a good approach. Here's how you can implement it:
&nbsp;
Use the Fabric API to monitor pipeline runs programmatically.Create a notebook activity in your pipeline that:Fetches the pipeline run status using the Fabric API.Logs the status (success/failure) for each pipeline and its child pipelines.Sends an email notification using an SMTP library (e.g., Python's smtplib) or a third-party email service like SendGrid.Parameterize the notebook to handle different pipelines dynamically.
&nbsp;
Using Webhooks for NotificationsYou can use webhooks to trigger notifications based on pipeline events.
&nbsp;
Configure a webhook in Microsoft Fabric to listen for pipeline run events (success/failure).Use a service like Azure Logic Apps, Power Automate, or a custom webhook handler to process the event and send email notifications.Ensure the webhook is triggered for both the master and child pipelines.
&nbsp;
Custom Logging and Notification PipelineCreate a separate pipeline dedicated to logging and notifications.
At the end of each pipeline (master and child), add an activity to log the run status to a centralized location (e.g., Azure Blob Storage, SQL Database).Trigger the logging and notification pipeline after the master pipeline completes.The logging pipeline reads the run statuses and sends email notifications.
&nbsp;
Using Azure Monitor AlertsLeverage Azure Monitor to track pipeline runs and send alerts.
Enable diagnostic settings for your pipelines to send logs to Azure Monitor.Create an alert rule in Azure Monitor to trigger on specific conditions (e.g., pipeline failure).Configure the alert to send email notifications.
&nbsp;
Combining Fabric API and Logic AppsIf you want a more robust solution, combine the Fabric API with Azure Logic Apps.
&nbsp;
Use the Fabric API to fetch pipeline run statuses.Trigger a Logic App workflow to process the statuses and send email notifications.Schedule the Logic App to run periodically or trigger it from the pipeline.
&nbsp;
Regards,
Nono Chen
If this post&nbsp;helps, then please consider&nbsp;Accept it as the solution&nbsp;to help the other members find it more quickly.

anonymous · Answer

Hi&nbsp;Anonymous&nbsp;,&nbsp;Thank you for the logic, I happen to use notebook at pipeline run time by using a wait activity(as the status might take some time to reflect since we are using API). I happen to use the notebook during pipeline run where I fetch my latest run. I am able to extract this. However, I am trying to use Outlook activity to push the notification. The issue is I am not sure how to convert my string output to Table.This is the structure of my notebook df output:Main Pipeline NameMaster Pipeline NamePipeline NameInvoke Pipeline NamePipeline IDStatusError MessagePL_Parameter_Main PipelineInvoke pipeline1PL_Parameter_xyz-abcFailed{Error_Message_will_be_displayed}PL_Parameter_PL_Parameter_Invoke pipeline1PL_Test_1efg-fhiFailed{Error_Message_will_be_displayed}I am using mssparkutils to capture this notebook output(exitValue) as a string and pass it to set variable(optional) then to Outlook activity. But since it is string, I am not sure how I can convert this string to table.Also, if we want to use SMTP, can I use FabricRestClient instead of passing username and password.Can you please share a sample structure for SMTP code as I don't want to pass username and password.Thanks in advance!Hrishi

anonymous · Answer

Also, we are only looking to integrate this as a complete Fabric solution (we don't want to use Azure).

anonymous · Answer

cant we use power automate for this ?&nbsp;

v-saisrao-msft · Answer

Hi&nbsp;Anonymous
Thanks for the detailed context&nbsp;you're on the right track with using a notebook and Fabric APIs. To convert your DataFrame output into a format suitable for the Outlook activity, I recommend using df.to_html(index=False) within your notebook to transform the DataFrame into an HTML table string. You can then pass this HTML string using mssparkutils.notebook.exit(), capture it in a Set Variable activity, and use it as the email body in your Outlook activity (assuming it supports HTML content). This approach maintains formatting and ensures readability in email notifications. Regarding SMTP, FabricRestClient currently doesn’t support credential-free SMTP directly. If avoiding username/password is critical, consider using the Outlook activity (if supported in your tenant) or triggering a webhook to Power Automate for notifications, which provides secure and flexible email delivery without handling credentials directly.
&nbsp;
If this post&nbsp;helps, then please give us ‘Kudos’ and consider&nbsp;Accept it as a solution&nbsp;to help the other members find it more quickly.
&nbsp;
Thank you.

Forum Discussion

Trigger Alerts Notification for successful/failed Pipelines

8 Replies

Related content

Pipeline Fails with "Operation returned an invalid status code 'BadRequest'" After Successful Run

Notifications on pipeline error

Not receiving Failure Notification on Fabric Pipelines and Notebooks

Schedule Failure Notification in Notebook and Data pipeline

Copy Activity Fails but shows as successful

Main Pipeline Name	Master Pipeline Name	Pipeline Name	Invoke Pipeline Name	Pipeline ID	Status	Error Message
PL_Parameter_	Main Pipeline	Invoke pipeline1	PL_Parameter_	xyz-abc	Failed	{Error_Message_will_be_displayed}
PL_Parameter_	PL_Parameter_	Invoke pipeline1	PL_Test_1	efg-fhi	Failed	{Error_Message_will_be_displayed}