Forum Discussion

jubinsoni's avatar
jubinsoni
Icon for Solution Sage rankSolution Sage
2 days ago

Error handling pattern for ForEach with partial failures: continue and report, or fail fast?

Hi everyone,

I have a pipeline that loops over a list of tables with ForEach and runs a Copy activity per item. When one table fails, I would like the remaining tables to still load, and to get one consolidated failure report at the end instead of the whole run failing on the first error.

The pattern I am considering:

  • Wrap the Copy activity so its failure path (the On fail dependency) logs the table name and error message to a control table, and the ForEach item itself finishes as succeeded.
  • After the loop, a Script or Lookup activity counts rows in the control table for this run, and a Fail activity raises one error with the summary if the count is above zero.

A sketch of the final check:

SELECT COUNT(*) AS failed_tables
FROM ctl.load_log
WHERE run_id = @{pipeline().RunId}
  AND status = 'Failed';

Questions:

  • Is this the usual approach, or is there a built-in way to get partial-success semantics for a ForEach?
  • How do you make sure the pipeline run status still reflects the failures so alerts fire?
  • Any gotchas with the sequential versus parallel setting when logging from inside the loop?

Thanks in advance.

pipelines

2 Replies

  • v-saisrao-msft's avatar
    v-saisrao-msft
    Icon for Community Support rankCommunity Support
    2 days ago

    Hi jubinsoni​ ,
    Try this, use a Try-Catch-Proceed pattern inside the ForEach activity. Configure the Copy activity's Upon Failure path to log the failed table details in a control table. Connect both Upon Failure and Upon Skip paths from the logging activity to the subsequent activity so processing can continue. After the ForEach completes, use a Lookup activity to retrieve the failure count for the current pipeline Run ID, followed by an If Condition to evaluate the result. If failures exist, execute a Fail activity with a meaningful error message; otherwise, allow the pipeline to complete successfully. For parallel execution, use independent failure records instead of shared variables for failure counting, and ensure the logging mechanism supports concurrent writes.
    Pipeline failure and error message - Azure Data Factory | Microsoft Learn
    ForEach Activity - Microsoft Fabric | Microsoft Learn
    Fail activity - Microsoft Fabric | Microsoft Learn
    Lookup activity - Microsoft Fabric | Microsoft Learn

    Thank you.


  • jubinsoni's avatar
    jubinsoni
    Icon for Solution Sage rankSolution Sage
    1 day ago

    Thank you v-saisrao-msft​. Here is the summary of the pattern from this thread.

    Short answer: let the loop continue past failures, record each failure, then raise one consolidated failure after the ForEach so the run status still shows red and alerts fire.

    Inside ForEach -> Copy activity; its Upon Failure path logs table name + error to a control table
Continue       -> connect both Upon Failure and Upon Skip from the logging activity to the next activity
After loop     -> Lookup: failure count for pipeline().RunId in the control table
Decision       -> If Condition: count > 0 -> Fail activity with a clear summary message; else succeed
Parallel runs  -> write independent failure rows instead of incrementing a shared variable;
                  make sure the logging target handles concurrent writes

    The final Fail activity is what keeps the pipeline run status accurate, so monitoring and alerts still work even though individual tables did not stop the loop.

    AI-assisted drafting: AI was used to help structure and phrase this response. I reviewed and validated the technical content before posting.