topic Re: How to Call Another Pipeline on Failure and Store Error Details in Lakehouse? in Pipelines

How to Call Another Pipeline on Failure and Store Error Details in Lakehouse?

bhavya5903 — Wed, 05 Feb 2025 10:25:33 GMT

I am working with Microsoft Fabric Data Pipelines and need to implement a failure-handling mechanism.

I have a main pipeline with multiple activities (e.g., Copy Data, Set Variable, etc.).
If any activity fails, I want to trigger another pipeline (FailureHandlerPipeline).
I need to pass failure details dynamically, including:
- FailedActivityName
- ErrorMessage
- ErrorCode
- PipelineName
The FailureHandlerPipeline should store these details in a Lakehouse table for logging and troubleshooting.

How do I dynamically capture failure details from multiple activities and pass them to the failure-handling pipeline?
What’s the best approach to store these details in Lakehouse using a low-code method?
Can I handle multiple failures at the same time? If multiple activities fail, will the failure-handling pipeline execute multiple times, or is there a way to batch the errors?

I appreciate any guidance, best practices, or alternative approaches!

NandanHegde — Wed, 05 Feb 2025 10:47:10 GMT

to manage error logging for multiple activities.

You can use the combination of on skip, on completion and on failure dependencies to trigger another pipeline on failure as stated in the aboe blog.

You can dynamically get the pipelinename, error messgae and error code via system variables but you cannot get the activity name dynamically

bhavya5903 — Wed, 05 Feb 2025 13:16:57 GMT

The FailureHandlerPipeline should store these details in a Lakehouse table for logging and troubleshooting

bhavya5903 — Mon, 10 Feb 2025 05:33:52 GMT

Can anyone help me with this?

NandanHegde — Mon, 10 Feb 2025 05:58:32 GMT

Sorry to have missed out on this thread.

You can refer the below blog wherein you can copy the varibles/activity outpput into lakehouse :