Forum Discussion

jubinsoni's avatar
jubinsoni
Icon for Solution Sage rankSolution Sage
3 days ago
Solved

Error handling pattern for ForEach with partial failures: continue and report, or fail fast?

Hi everyone, I have a pipeline that loops over a list of tables with ForEach and runs a Copy activity per item. When one table fails, I would like the remaining tables to still load, and to get one ...
  • jubinsoni's avatar
    2 days ago

    Thank you v-saisrao-msft​. Here is the summary of the pattern from this thread.

    Short answer: let the loop continue past failures, record each failure, then raise one consolidated failure after the ForEach so the run status still shows red and alerts fire.

    Inside ForEach -> Copy activity; its Upon Failure path logs table name + error to a control table
    Continue       -> connect both Upon Failure and Upon Skip from the logging activity to the next activity
    After loop     -> Lookup: failure count for pipeline().RunId in the control table
    Decision       -> If Condition: count > 0 -> Fail activity with a clear summary message; else succeed
    Parallel runs  -> write independent failure rows instead of incrementing a shared variable;
                      make sure the logging target handles concurrent writes

    The final Fail activity is what keeps the pipeline run status accurate, so monitoring and alerts still work even though individual tables did not stop the loop.

    AI-assisted drafting: AI was used to help structure and phrase this response. I reviewed and validated the technical content before posting.