QUESTION::PIPELINE::SET VARIABLE::RACE CONDITION

[Element115]

Let's say you have a situation like this one:

and there is only one variable, meaning the variable being referenced by both Set variable activities is the same. And after the Set variable, to the right (not shown here), the next activity, say a IF condition, depends on the value of that one variable.

Can the following scenario happen? --> after the top Set variable completes can the bottom Set variable also complete BEFORE the value of the variable reaches the next activity in the top path (stuff on the right that is not shown here)?

Let's say the activity that comes after the Set variable in the top path is a condition IF statement that expects a value of 1 and let's say that the activity that comes after the Set variable in the bottom path is a condition IF statement that expects a value of 2.

If top completes first, then the variable value = 1.  Then the bottom path completes, and the variable value = 2. Then the execution of the top path reaches a condition testing for IF variable = 1. But since the bottom path changed the value of the variable to 2, the top condition in the top path with evalute to false instead of true (remember that the top path Set variable was 1).

Is this scenario possible?  And if yes, is there a mechanism by which we can specify how the pipeline variables should be accessed , ie no shared memory, or some lock mechanism, etc..., instead of having to define 2 different variable names?

[Element115]

@v-nikhilan-msft  Could someone provide some pointers re this question please?

[v-nikhilan-msft]

Hi @Element115 

Thanks for using Fabric Community. Apologies for the delay in response.

At this time, we are reaching out to the internal team to get some help on this. We will update you once we hear back from them.
Thanks 

[v-nikhilan-msft · Accepted solution]

Hi @Element115 
Yes, the scenario you described is possible. In some pipeline environments, activities can run concurrently by default, which means the set variable activities could update the same variable at nearly the same time. But this would be a bad design. This could lead to unexpected results, where the IF condition evaluates to the wrong value.

In the scenario you described, the top path sets the variable to 1 and then the bottom path sets the variable to 2 before the top path’s IF condition activity gets to check the variable. This would indeed cause the IF condition in the top path to evaluate to false instead of true.

To avoid this, you could use two different variables for the two paths. This way, each path has its own variable and there’s no risk of one path overwriting the variable’s value before the other path gets to check it.

Alternatively, if you want to use a single variable but ensure that its value isn’t overwritten by another activity before it’s checked, you would need some kind of locking mechanism. However, such a mechanism is not typically available in most data pipeline systems. The best practice in such cases is to design the pipeline in such a way that shared variables are not being written to by multiple activities that could potentially run in parallel.

Use two different variables and within the IF conditional, if you need to check a value of one of the other, can use dynamic expression. Variables are global within the pipeline run.
In the above screenshot, there is no way to guarantee which Storedproc + setVar combination will finish first.

Hope this helps. Please let me know if you have any further questions.