Forum Discussion
QUESTION::PIPELINE::SET VARIABLE::RACE CONDITION
Hi Element115
Yes, the scenario you described is possible. In some pipeline environments, activities can run concurrently by default, which means the set variable activities could update the same variable at nearly the same time. But this would be a bad design. This could lead to unexpected results, where the IF condition evaluates to the wrong value.In the scenario you described, the top path sets the variable to 1 and then the bottom path sets the variable to 2 before the top path’s IF condition activity gets to check the variable. This would indeed cause the IF condition in the top path to evaluate to false instead of true.
To avoid this, you could use two different variables for the two paths. This way, each path has its own variable and there’s no risk of one path overwriting the variable’s value before the other path gets to check it.
Alternatively, if you want to use a single variable but ensure that its value isn’t overwritten by another activity before it’s checked, you would need some kind of locking mechanism. However, such a mechanism is not typically available in most data pipeline systems. The best practice in such cases is to design the pipeline in such a way that shared variables are not being written to by multiple activities that could potentially run in parallel.
Use two different variables and within the IF conditional, if you need to check a value of one of the other, can use dynamic expression. Variables are global within the pipeline run.
In the above screenshot, there is no way to guarantee which Storedproc + setVar combination will finish first.
Hope this helps. Please let me know if you have any further questions.
Hi Element115
Thanks for using Fabric Community. Apologies for the delay in response.
At this time, we are reaching out to the internal team to get some help on this. We will update you once we hear back from them.
Thanks
Hi Element115
Yes, the scenario you described is possible. In some pipeline environments, activities can run concurrently by default, which means the set variable activities could update the same variable at nearly the same time. But this would be a bad design. This could lead to unexpected results, where the IF condition evaluates to the wrong value.
In the scenario you described, the top path sets the variable to 1 and then the bottom path sets the variable to 2 before the top path’s IF condition activity gets to check the variable. This would indeed cause the IF condition in the top path to evaluate to false instead of true.
To avoid this, you could use two different variables for the two paths. This way, each path has its own variable and there’s no risk of one path overwriting the variable’s value before the other path gets to check it.
Alternatively, if you want to use a single variable but ensure that its value isn’t overwritten by another activity before it’s checked, you would need some kind of locking mechanism. However, such a mechanism is not typically available in most data pipeline systems. The best practice in such cases is to design the pipeline in such a way that shared variables are not being written to by multiple activities that could potentially run in parallel.
Use two different variables and within the IF conditional, if you need to check a value of one of the other, can use dynamic expression. Variables are global within the pipeline run.
In the above screenshot, there is no way to guarantee which Storedproc + setVar combination will finish first.
Hope this helps. Please let me know if you have any further questions.