Parallel Processing with For-Each Activity in Fabric - Fabric Data Factory Pipeline

[uselessai_in]

Hi all, I recently faced an issue with parallel processing when using for each loop with set variable activity.

The issue was "Set Variable" activity forces you to do processing sequentially, which'll take alot of time when processing mutiple files. 

I recently got one solution for fixing this issue. Read more about the solution I found here  - Parallel Processing in Fabric

Now the question is what are the other strategies that one can use for parallel processing in Fabric's Data Factory. 

[v-pnaroju-msft]

Hi uselessai_in,

We wanted to check in regarding your query, as we have not heard back from you. If you found our response useful, kindly mark it as the accepted solution and provide kudos to guide other members.

Thank you.

[v-pnaroju-msft]

Hi uselessai_in,

We are following up on your query, as we have not received a response from you. If our previous response was helpful, please mark it as the accepted solution and provide kudos to assist other members.

Thank you.

[v-pnaroju-msft]

Hi uselessai_in,

We wanted to check in regarding your query, as we have not heard back from you. If you found our response useful, kindly mark it as the accepted solution and provide kudos to guide other members.

Thank you.

[v-pnaroju-msft · Accepted solution]

Hi uselessai_in,

We appreciate your inquiry on the Microsoft Fabric Community Forum.

Thank you for sharing your solution. Kindly find below an alternative approach for parallel processing in Fabric’s Data Factory:

You may utilise the Copy Activity with the parallelCopies property to copy multiple files simultaneously.

For further reference, please find the following links:
How to copy data using copy activity - Microsoft Fabric | Microsoft Learn
Activity overview - Microsoft Fabric | Microsoft Learn

If you find this response helpful, kindly mark it as the accepted solution and provide kudos. This will assist other community members who may have similar queries.

Thank you.

[uselessai_in]

If you found the above solution helpful. Drop a like and share other strategies.