Forum Discussion
Helper IIParallel Processing with For-Each Activity in Fabric - Fabric Data Factory Pipeline
Hi all, I recently faced an issue with parallel processing when using for each loop with set variable activity.
The issue was "Set Variable" activity forces you to do processing sequentially, which'll take alot of time when processing mutiple files.
I recently got one solution for fixing this issue. Read more about the solution I found here - Parallel Processing in Fabric
Now the question is what are the other strategies that one can use for parallel processing in Fabric's Data Factory.
Hi uselessai_in,
We appreciate your inquiry on the Microsoft Fabric Community Forum.
Thank you for sharing your solution. Kindly find below an alternative approach for parallel processing in Fabric’s Data Factory:
You may utilise the Copy Activity with the parallelCopies property to copy multiple files simultaneously.
For further reference, please find the following links:
How to copy data using copy activity - Microsoft Fabric | Microsoft Learn
Activity overview - Microsoft Fabric | Microsoft LearnIf you find this response helpful, kindly mark it as the accepted solution and provide kudos. This will assist other community members who may have similar queries.
Thank you.
5 Replies
uselessai_inIf you found the above solution helpful. Drop a like and share other strategies.
- v-pnaroju-msft
Community Support
Hi uselessai_in,
We appreciate your inquiry on the Microsoft Fabric Community Forum.
Thank you for sharing your solution. Kindly find below an alternative approach for parallel processing in Fabric’s Data Factory:
You may utilise the Copy Activity with the parallelCopies property to copy multiple files simultaneously.
For further reference, please find the following links:
How to copy data using copy activity - Microsoft Fabric | Microsoft Learn
Activity overview - Microsoft Fabric | Microsoft LearnIf you find this response helpful, kindly mark it as the accepted solution and provide kudos. This will assist other community members who may have similar queries.
Thank you.
- v-pnaroju-msft
Hi uselessai_in,
We wanted to check in regarding your query, as we have not heard back from you. If you found our response useful, kindly mark it as the accepted solution and provide kudos to guide other members.
Thank you.
- v-pnaroju-msft
Hi uselessai_in,
We are following up on your query, as we have not received a response from you. If our previous response was helpful, please mark it as the accepted solution and provide kudos to assist other members.
Thank you.
- v-pnaroju-msft
Hi uselessai_in,
We wanted to check in regarding your query, as we have not heard back from you. If you found our response useful, kindly mark it as the accepted solution and provide kudos to guide other members.
Thank you.