topic Re: Data Pipeline failure alert in Pipelines

Data Pipeline failure alert

Srisakthi — Fri, 15 Nov 2024 10:37:48 GMT

Hello Everyone,

I have tried adding MSteams activity when one of my activity fails. But is there anyway to alert when my pipeline fails?

Is there anyway to alert my data engineers when my pipeline fails?

Regards,

Srisakthi

Re: Data Pipeline failure alert

SudhavaniKolla3 — Fri, 15 Nov 2024 17:10:23 GMT

Hi, you can use Outlook

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

Re: Data Pipeline failure alert

spencer_sa — Fri, 15 Nov 2024 23:58:42 GMT

In addition to sending an Outlook message (as SudhavaniKolla3 says) you can also use the Teams activity to send a message to a Teams group or channel.

Be aware, using the OnFail flow anchors in pipeline activities is akin to a try: exception: block in that the main pipeline will no longer fail on that activity failing, so you'll need to add an explicit 'Fail' activity to force the main pipeline to fail.

Re: Data Pipeline failure alert

Srisakthi — Mon, 18 Nov 2024 04:01:24 GMT

Hello,

I know that we can an activity to my pipeline and use OnFail action to take further actions by sending it to Outlook, teams or even i can use web activity to log it somewhere. Here the questions suppose i have 10 activity in my pipeline , do I need to add OnFail for each activity? I want to add only at one place and send a notification.

I found one way which is to add the respective pipeline in another pipeline(and use invoke pipeline) and attach to notification activity. Is there any other approach?

Regards,

Srisakthi

Re: Data Pipeline failure alert

NandanHegde — Tue, 19 Nov 2024 12:17:27 GMT

Unfortunately, unlike ADF wherein you can have in built alerts to notify in case of pipeline failures; in fabric data pipelines you have to explictily manage the failure notifications yourself.

Rather than having team/outlook activity after each activity, you can use the below method :

https://www.c-sharpcorner.com/article/error-logging-and-the-art-of-avoiding-redundant-activities-in-msft-fabric-data-p/

Re: Data Pipeline failure alert

SudhavaniKolla3 — Wed, 20 Nov 2024 05:23:03 GMT

you can add all activities and conenct them on success of first activity, second activity shoud run and on success of second activity, 3rd activity should run, like this you have to attach first, and at the end of last activity, you should add teams or outlook activity on failure of last activity.

so that if any activty fails then pipeline also gets fails, then you will get mail notification.

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

Re: Data Pipeline failure alert

NandanHegde — Wed, 20 Nov 2024 05:22:00 GMT

Plz refer this :

https://www.c-sharpcorner.com/article/error-logging-and-the-art-of-avoiding-redundant-activities-in-msft-fabric-data-p/

SudhavaniKolla3 The above flow that you have created wont work in case if there is any failure in lookup or copy or dataflow activity. plz refer the above blog for the correct flow creation