How to avoid a nested ForEach activity and a json settings file

eurenergy — Thu, 06 Mar 2025 10:17:46 GMT

I have the pipeline belwo which works great. It works for a single process and uses paths etc for this process.

However, I need to use this pipeline for five processes, (in which some settings are different every run, like the path it takes files from, the file type, and the destination tables).

So my solution:

Add a lookup activity which looks up a json file stored in my lakehouse. This json file has all the settings for each of the processes:

Then: I would copy the pipeline mentioned earlier as a whole in a ForEach activity, so the correct settings are applied on each run. This would like like the below:

However, the problem is that in data factory, it is not possible to use a ForEach activity within a ForEach activity.... And I do really need these ForEach activity's to copy multiple files etc.

So my question is: how to do this in an elegant, dynamic way?

A solution would be to copy the whole pipeline, paste it 5 times under each other. But then if I need to make a change i need to do it 5 times.

Re: How to avoid a nested ForEach activity and a json settings file

NandanHegde — Thu, 06 Mar 2025 11:37:43 GMT

you can trigger an Execute pipeline activity from within a for each activity.

So you can have a pipeline with your main framework and call that pipeline from within foreach activity in another pipeline

topic How to avoid a nested ForEach activity and a json settings file in Pipelines

How to avoid a nested ForEach activity and a json settings file

Re: How to avoid a nested ForEach activity and a json settings file