Monitor pipeline runs with alerts

[GammaRamma]

We use the Monitor in Fabric to track pipeline runs. We do integration of on prem data within these pipelines among other things and there is always a risk of failures. So we need to have a comprehensive monitoring solution.

We have not been able to find a solution for two things :

1. Receive alerts on failed runs

2. Create monitoring dashboards that are visible to the team

We have looked into the Real-Time Hub but haven't found a solution there.

Does anyone know of a solution?

Thanks

[JoanAmal]

We run multiple scheduled pipelines, each with chained Invoke pipelines. How to consolidate multiple pipeline failures into a single monitoring system? How can we monitor all of them together and send one Teams alert when any pipeline fails? Please give step-by-step instructions. 

[Anonymous · Accepted solution]

Hi @GammaRamma ,

The following alternatives can be tried:

• Create a pipeline without any notification activity.
• Create a second empty pipeline and add an Invoke Pipeline to invoke the first pipeline.
• Add Outlook activity to the “On fail” node of the Invoke Pipeline activity.

When it fails, you will be notified by email. Just like below:

Best Regards,
Adamk Kong

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

[FabianSchut]

Hi @GammaRamma,

You could use a combination of a logic app and a web activity in Fabric data pipeline to achieve this. You should connect the failed line of a pipeline activity to the web activity that makes an http request to your Logic app. That Logic app will send an e-mail notifcying you of your failed pipeline. An instruction to set this up in Azure Data Factory can be found here and can be used in MS Fabric too: https://learn.microsoft.com/en-us/azure/data-factory/how-to-send-email.

For monitoring dashboards, you could create some more advanced logging. An example is to use a copy activity with a custom query to select your specific logging needed. In that query, you could take for example name and run id of the pipeline activity. The destination of that copy activity could be your own lakehouse table with all desired columns for your logging. Based on that table, you could create a Power BI dashboard.

The advantage of having a Power BI dashboard is that you could now use the Data Activator from Fabric to alert people when a pipeline has failed. The documentation for setting up this alert can be found here: https://learn.microsoft.com/en-us/fabric/data-activator/data-activator-create-triggers-design-mode. Then, you don't need the Logic App anymore.

Kind regards,

Fabian

[Srisakthi]

Hi @FabianSchut ,

I could not see any option to send an alert from Data Activator when pipeline fails. Can you please point me to the proper url

Regards,

Srisakthi