Forum Discussion
Error handling pattern for ForEach with partial failures: continue and report, or fail fast?
- 2 days ago
Thank you v-saisrao-msft. Here is the summary of the pattern from this thread.
Short answer: let the loop continue past failures, record each failure, then raise one consolidated failure after the ForEach so the run status still shows red and alerts fire.
Inside ForEach -> Copy activity; its Upon Failure path logs table name + error to a control table Continue -> connect both Upon Failure and Upon Skip from the logging activity to the next activity After loop -> Lookup: failure count for pipeline().RunId in the control table Decision -> If Condition: count > 0 -> Fail activity with a clear summary message; else succeed Parallel runs -> write independent failure rows instead of incrementing a shared variable; make sure the logging target handles concurrent writesThe final Fail activity is what keeps the pipeline run status accurate, so monitoring and alerts still work even though individual tables did not stop the loop.
AI-assisted drafting: AI was used to help structure and phrase this response. I reviewed and validated the technical content before posting.
Hi jubinsoni ,
Try this, use a Try-Catch-Proceed pattern inside the ForEach activity. Configure the Copy activity's Upon Failure path to log the failed table details in a control table. Connect both Upon Failure and Upon Skip paths from the logging activity to the subsequent activity so processing can continue. After the ForEach completes, use a Lookup activity to retrieve the failure count for the current pipeline Run ID, followed by an If Condition to evaluate the result. If failures exist, execute a Fail activity with a meaningful error message; otherwise, allow the pipeline to complete successfully. For parallel execution, use independent failure records instead of shared variables for failure counting, and ensure the logging mechanism supports concurrent writes.
Pipeline failure and error message - Azure Data Factory | Microsoft Learn
ForEach Activity - Microsoft Fabric | Microsoft Learn
Fail activity - Microsoft Fabric | Microsoft Learn
Lookup activity - Microsoft Fabric | Microsoft Learn
Thank you.