How to Call Another Pipeline on Failure and Store Error Details in Lakehouse?

[bhavya5903]

Question:

I am working with Microsoft Fabric Data Pipelines and need to implement a failure-handling mechanism.

Scenario:

• I have a main pipeline with multiple activities (e.g., Copy Data, Set Variable, etc.).
• If any activity fails, I want to trigger another pipeline (FailureHandlerPipeline).
• I need to pass failure details dynamically, including:
  • FailedActivityName
  • ErrorMessage
  • ErrorCode
  • PipelineName
• The FailureHandlerPipeline should store these details in a Lakehouse table for logging and troubleshooting.

Challenges:

1. How do I dynamically capture failure details from multiple activities and pass them to the failure-handling pipeline?
2. What’s the best approach to store these details in Lakehouse using a low-code method?
3. Can I handle multiple failures at the same time? If multiple activities fail, will the failure-handling pipeline execute multiple times, or is there a way to batch the errors?

I appreciate any guidance, best practices, or alternative approaches!

[bhavya5903]

Can anyone help me with this?

[NandanHegde · Accepted solution]

Sorry to have missed out on this thread.

You can refer the below blog wherein you can copy the varibles/activity outpput into lakehouse :

https://datasharkx.wordpress.com/2023/04/20/copying-variable-parameter-into-sink-azure-blob-storage-...

[NandanHegde]

please refer this blog : https://datasharkx.wordpress.com/2021/08/19/error-logging-and-the-art-of-avoiding-redundant-activiti... 

to manage error logging for multiple activities.

You can use the combination of on skip, on completion and on failure dependencies to trigger another pipeline on failure as stated in the aboe blog.

You can dynamically get the pipelinename, error messgae and error code via system variables but you cannot get the activity name dynamically

[bhavya5903]

The FailureHandlerPipeline should store these details in a Lakehouse table for logging and troubleshooting