topic Re: Email on Error in Pipeline in Pipelines

Email on Error in Pipeline

bcdobbs — Tue, 05 Dec 2023 09:40:54 GMT

I've built a pipeline that runs two notebooks and I'm wanting it to email on failure.

It looks like:

With the Send Error Email connected on the fail nodes.

However it never triggers the outlook task:

Any suggestion on what I'm doing wrong or is there a better way to handle this?

Thanks

Ben

Re: Email on Error in Pipeline

bcdobbs — Tue, 05 Dec 2023 09:57:33 GMT

Having looked at the JSON for the pipeline I now realise the way I'd built it would only trigger an error if both notebooks failed which couldn't happen in that setup.

I've added separate notification blocks and it now works.

Is there however a cleaner way of doing this? Eg trigger an email on any failure anywhere in the pipeline?

Re: Email on Error in Pipeline

Anonymous — Tue, 05 Dec 2023 10:22:34 GMT

Hi bcdobbs ,

Thanks for using Fabric Community.

As I tried to repro your scenario I have noticed this.

Case 1: Every thing worked well

Case 2: I can find the same issue you are facing

For time being try to link to only one Notebook Activity on failure.

In meantime I will check with internal to get some information on this behaviour.

Thank you

Re: Email on Error in Pipeline

NandanHegde — Tue, 05 Dec 2023 11:40:55 GMT

Hey,

You right fully identified the cause that the email would be triggered only if both the notebook activities would fail as the expressions are AND scenario and that would never occur as the 2nd notebook would be trigerred only with the success of the 1st one.

So as a solution, you can have seperate notifications per activity but there is only a limitaion of 40 activities per pipeline.

So below blog though ADF related would have the same concpets for Fabric:

https://datasharkx.wordpress.com/2021/08/19/error-logging-and-the-art-of-avoiding-redundant-activities-in-azure-data-factory/

which would be a cleaner way

Re: Email on Error in Pipeline

Anonymous — Tue, 05 Dec 2023 14:45:11 GMT

Yes NandanHegde got it.

It looks like we don't have any issue with data factory, it is considered as a AND operator, not a OR operator.

bcdobbs I was able to trigger an email by creating a pipeline design like above.

Re: Email on Error in Pipeline

Anonymous — Thu, 07 Dec 2023 10:08:50 GMT

Hi bcdobbs ,

We haven’t heard from you on the last response and was just checking back to see if your query was answered. Otherwise, will respond back with the more details and we will try to help .

Re: Email on Error in Pipeline

Anonymous — Mon, 11 Dec 2023 10:41:03 GMT

Re: Email on Error in Pipeline

NandanHegde — Thu, 21 Dec 2023 08:37:50 GMT

hey bcdobbs

Are there any queries w.r.t creating a flow within the pipelines ?

Re: Email on Error in Pipeline

S_Rahlf — Thu, 18 Apr 2024 15:55:29 GMT

I was checking that out myself before stumbling upon this community thread as it is "an old friend from data factory" -> all "inbound" constraints are combined AND so a mulitple "on failure" from different activities will never run.

Our workaround, as a "sequence container" is also not available, is to use a sub pipeline returning in case of a failure in any activity a "on failure" from the "invoke pipeline" activity or as kind of "dirty hack" an "if condition" as sequence having a "always true" condition and grouping activites...

Re: Email on Error in Pipeline

Anonymous — Tue, 30 Jul 2024 16:58:00 GMT

The cleaner solution would be to wrap your pipeline within another pipeline, using the Invoke Pipeline activity.

Create your pipeline without any notification activities.
Create a second empty pipeline, and add Invoke Pipeline to call your first pipeline.
Add an Outlook activity to the "On fail" node of the Invoke Pipeline activity.

This will trigger the email if any part of your inner pipeline fails.

Re: Email on Error in Pipeline

smpa01 — Sat, 03 Aug 2024 05:19:07 GMT

bcdobbs I belive this would be possible.

You need to declare a global variable which is an empty array.

Then you need to create an Append Variable activity that you would connect to each of the activities in the pipeline and you would keep on appending the succees/fail status in the append variable. At t-1 step, you would create a if condition to check if the collection (array) contains any fail using contains. If yes, send an email. I did not give it a try though. Let me know if it works.