For Each in a Data Pipeline

[p_da]

Hey folks, 

I'm facing an issue with a ForEach in a Data pipeline. I have configured the For Each to run in parallel (maximum 3). The ForEach has as input a Lookup activity where there is only unique values per ID. However, I have noticed that in the ForEach activity it is running twice or more for the same unique ID. Does anyone knows how can I solve it ?

[Anonymous · Accepted solution]

Hi @p_da ,

I think you can try these two steps below:

1. Make sure that the input array of the ForEach activity does not contain duplicate values. Sometimes, due to joins or other operations, the Find activity may return duplicate rows.
2. since ForEach is set to run in up to 3 parallel runs, make sure that the isSequential property is set to false. this setting allows for parallel execution, but can sometimes lead to unexpected behavior if not properly configured. As a temporary solution, you can set the isSequential property to true and see if the problem persists.

If you want more information, you can check out this official document: ForEach activity - Azure Data Factory & Azure Synapse | Microsoft Learn

Best Regards

Yilong Zhou

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

[jwinchell40]

@p_da - Is it possible for you to show/share the logging so we can see this in action?  From the description itself it is hard to offer up anything meaningful.