Forum Discussion
Filter activity not working properly
- 1 year ago
The scope of a variable is at the pipeline level, not in the iteration of a foreach loop.
Thinking out loud (not tested this), I assume that the absolute simplest fix would be to set the foreach to sequential, but in most circumstances that's not desired outcome.
But to properly achieve this in paralleld, how I tend to approach this is produce a payload for the foreach that includes the translations rather that retrieving them separately through lookups withi the foreach. The foreach will naturally filter the set leaving you with the translations for the individual tables. You also have less contact with the database that way (i.e. you don't have lots of individual lookups happening concurrently).
Without seeing the wider solution, I'm making somes assumptions but if at some point you have a lookup that is retrieving tables to process, extending that with the mappings would solve this.
- 1 year ago
Thanks, justinjbird . That sequential run solution is correct, i have already tested it but since there are very large tables to pull, it is not ideal solution. when we look at the time when set variable activities are evaluated it is 11:08:26. But for filter it is 11:08:35. it means whatever value last set variable activity set for the variable, filter will only evaluate for that variable and will always give same output for every for loop iteration (unless again set variable activity run after filter and change variable name). I had unified mapping file for all tables. i divided that into seperate files for each table and now instead of searching for specific table mapping in a large file, i am pulling table mapping directly from its mapping file using tablename and its working fine.
Thanks, justinjbird . That sequential run solution is correct, i have already tested it but since there are very large tables to pull, it is not ideal solution. when we look at the time when set variable activities are evaluated it is 11:08:26. But for filter it is 11:08:35. it means whatever value last set variable activity set for the variable, filter will only evaluate for that variable and will always give same output for every for loop iteration (unless again set variable activity run after filter and change variable name). I had unified mapping file for all tables. i divided that into seperate files for each table and now instead of searching for specific table mapping in a large file, i am pulling table mapping directly from its mapping file using tablename and its working fine.
Hi ketan5050 ,
Thank you for sharing your findings and explaining the timestamp analysis it was an effective way to confirm the race condition caused by parallel execution.
You are correct- in parallel ForEach loops, pipeline variables can be overwritten before activities like Filter run, which explains the repeated translator output. I also appreciate your workaround of dividing the unified mapping file into separate files for each table and referencing them by table name. This approach helps prevent variable conflicts, simplifies the logic, and can improve performance by reducing concurrent lookups.
Thanks for your prompt response justinjbird .
Regards,
Yugandhar.