topic Pipeline doesn't execute On Error in Data Engineering

Pipeline doesn't execute On Error

lsabetta — Mon, 06 Oct 2025 15:03:15 GMT

Hi community,

My pipeline is not running the notebook that has to run in case one of my activities fail. It stops just after the activity that has failed. I don't know if it is related but all the 3 notebooks run within the same Session Tag
Does anyone know why?

Re: Pipeline doesn't execute On Error

apturlov — Mon, 06 Oct 2025 16:04:52 GMT

@lsabetta could you provide the output from a failed activity? It's also not obvious from your screenshot which activity actually failed.

Re: Pipeline doesn't execute On Error

tayloramy — Tue, 07 Oct 2025 01:38:06 GMT

Hi @lsabetta,

The connections in a pipeline are treated as AND connections, so your nb_finish_error won't run unless all 4 other activities fail.

So the behaviour you're seeing is expected.

To have OR logic, you will need to use variables and build some custom logic.

If you found this helpful, consider giving some Kudos. If I answered your question or solved your problem, mark this post as the solution.

Re: Pipeline doesn't execute On Error

smeetsh — Tue, 07 Oct 2025 02:52:59 GMT

Sadly the "on fail" is an AND and not an OR. In your case the notebook will only trigger if all parent activities fail and not just one of them.

This means you will have to create a notebook for each fail activity, which is indeed a pain if you have many activities in one pipeline. We created something similar, but with using the outlook activity to send an email if a specific step fails.

I have looked at using a notebook to just look at the whole pipeline and alert if the pipeline itself fails, but there are 2 downsides to that.

1: It won't give any detail, but the alert would tell you to go look at the log, since something is wrong

2: I would need to set up an api-key etc, which I haven't done yet, and it would create more admin work

Hopefully this is of use to you.

Cheers

Hans

(If my answer is helpful give it a kudo and/or mark it as a solution)