Email on Error in Pipeline

[bcdobbs]

I've built a pipeline that runs two notebooks and I'm wanting it to email on failure.

It looks like:

With the Send Error Email connected on the fail nodes.

However it never triggers the outlook task:

Any suggestion on what I'm doing wrong or is there a better way to handle this?

Thanks

Ben

[smpa01]

@bcdobbs I belive this would be possible.

You need to declare a global variable which is an empty array.

Then you need to create an Append Variable activity that you would connect to each of the activities in the pipeline and you would keep on appending the succees/fail status in the append variable. At t-1 step, you would create a if condition to check if the collection (array) contains any fail using contains. If yes, send an email. I did not give it a try though. Let me know if it works.

[SeanTavares]

The cleaner solution would be to wrap your pipeline within another pipeline, using the Invoke Pipeline activity.

1. Create your pipeline without any notification activities.
2. Create a second empty pipeline, and add Invoke Pipeline to call your first pipeline.
3. Add an Outlook activity to the "On fail" node of the Invoke Pipeline activity.

This will trigger the email if any part of your inner pipeline fails.

[S_Rahlf]

I was checking that out myself before stumbling upon this community thread as it is "an old friend from data factory" -> all "inbound" constraints are combined AND so a mulitple "on failure" from different activities will never run.

Our workaround, as a "sequence container" is also not available, is to use a sub pipeline returning in case of a failure in any activity a "on failure" from the "invoke pipeline" activity or as kind of "dirty hack" an "if condition" as sequence having a "always true" condition and grouping activites...

[NandanHegde]

Hey,

You right fully identified the cause that the email would be triggered only if both the notebook activities would fail as the expressions are AND scenario and that would never occur as the 2nd notebook would be trigerred only with the success of the 1st one.

So as a solution, you can have seperate notifications per activity but there is only a limitaion of 40 activities per pipeline.

So below blog though ADF related would have the same concpets for Fabric:

https://datasharkx.wordpress.com/2021/08/19/error-logging-and-the-art-of-avoiding-redundant-activiti...

which would be a cleaner way

[Anonymous · Accepted solution]

Yes @NandanHegde got it.

It looks like we don't have any issue with data factory, it is considered as a AND operator, not a OR operator.

@bcdobbs  I was able to trigger an email by creating a pipeline design like above.

[Anonymous]

Hi @bcdobbs ,

We haven’t heard from you on the last response and was just checking back to see if your query was answered. Otherwise, will respond back with the more details and we will try to help .

[Anonymous]

Hi @bcdobbs ,

We haven’t heard from you on the last response and was just checking back to see if your query was answered. Otherwise, will respond back with the more details and we will try to help .

[NandanHegde]

hey @bcdobbs 

Are there any queries w.r.t creating a flow within the pipelines ?

[Anonymous]

Hi @bcdobbs ,

Thanks for using Fabric Community.

As I tried to repro your scenario I have noticed this.

Case 1: Every thing worked well

Case 2: I can find the same issue you are facing

For time being try to link to only one Notebook Activity on failure.

In meantime I will check with internal to get some information on this behaviour.

Thank you

[bcdobbs]

Having looked at the JSON for the pipeline I now realise the way I'd built it would only trigger an error if both notebooks failed which couldn't happen in that setup.

I've added separate notification blocks and it now works.

Is there however a cleaner way of doing this? Eg trigger an email on any failure anywhere in the pipeline?