Pipeline ForEach Activity parallel execution interferes with Set Variable

[Anonymous]

Hello everyone!

I have a pipeline like this:

If I set the ForEach activity to sequential execution, everything works fine.

However, if I set it to parallel, the variable gets set twice first and then the copy activity is executed twice for the last set variable value (I know this isn't visible in the picture, I just want to illustrate it).

This is not what I understand as parallelism; this is sequentialism with a surprise order of operations. Is this happening to anyone else? What is going on? Can I ameliorate it? Thanks for your input!

[tmjones2]

It would be really neat if they could give us a warning when you set a variable inside of a parallel for each. But since they don't, we all get to waste an hour trying to figure out what is going on for ourselves.

Cool.

[Anonymous]

By the way, how I solved this now is to not even work with a variable. I directly use the ForEach item and subselect the column I want like this: @item().column_name

[Anonymous · Accepted solution]

Hi @Anonymous ,

When running in parallel, variables can indeed behave unpredictably because they are not thread-safe. This can lead to the kind of unexpected behavior you’re seeing, where the variable gets set multiple times before the copy activity executes.

Since variables are not thread-safe, try to avoid using them within parallel loops. Instead, consider using parameters or passing values directly to activities.

If possible, stick with sequential execution for activities that depend on variable values. This ensures that each iteration completes before the next one starts, maintaining the correct order of operations.

Also I think you can look at this document: Running a ForEach activity in parallel mode - Microsoft Q&A

Best Regards

Yilong Zhou

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.