topic Re: Pushing inner forEach activities to an outer variable in Pipelines

Pushing inner forEach activities to an outer variable

smpa01 — Thu, 25 Jul 2024 17:25:44 GMT

Is there any way to push the output value of forEach activity to an outer variable?

E.g.

For example,

Lookup is a SQL look up that returns following and Filter activity filters based on filter_action = 'include'

// output value Lookup { "count": 2, "value": [ { "id": 1, "filter_action": "include" }, { "id": 2, "filter_action": "include" }, { "id": 3, "filter_action": "exclude" } ] } //output value Filter { "ItemsCount": 2, "FilteredItemsCount": 2, "Value": [ { "id": 1, "filter_action": "include" }, { "id": 2, "filter_action": "include" } ] }

I am storing the output of filter in a variable and based on the array in Filter I am doing Copy Activity forEach objects present in Value array of Filter.

While the copying is taking place forEach ingestion how can I access the outer Variable to copy the output of Copy Activity in the same variable.

Equivalent Javascript code of what I am hoping to achieve in pipeline

Is it doable?

Re: Pushing inner forEach activities to an outer variable

Anonymous — Fri, 26 Jul 2024 02:12:24 GMT

Hi smpa01 ,

Currently Data Factory supports Run a Script activity and send an output summary by email.

I think you can create a Data pipeline and then select Script in the Activities.

Then break it down in detail based on the formatting that needs to be changed.

Eventually it would be possible to push the internal forEach activity to an external email as a Script.

Best Regards

Yilong Zhou

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

Re: Pushing inner forEach activities to an outer variable

frithjof_v — Fri, 26 Jul 2024 10:30:16 GMT

If your variable is of type 'Array', then the 'Append variable' activity can be used inside each foreach iteration to dynamically append new items to the array variable.

Append variable activity - Microsoft Fabric | Microsoft Learn

Afterwards, the array variable (with the data from the foreach activity) will be accessible for subsequent pipeline activities outside of the foreach activity.

I think the Collection functions are useful for accessing the data from the array variable in subsequent steps:

Expressions and functions - Microsoft Fabric | Microsoft Learn

Also other functions and activities can be useful.