Currently, in Data Factory, the ForEach activity requires a static batch count when executing in parallel mode. However, this fixed value does not always align with dynamic workloads, leading to inefficiencies. A more flexible approach would be to allow the batch count to be dynamic, adjusting automatically based on the number of items in the loop or available system resources.
For example, instead of manually setting a batch count of 10, users could define a dynamic value based on the total number of items, ensuring optimal parallel execution without overloading the system. This could be achieved by allowing expressions, pipeline parameters, or metadata-driven configurations to determine batch size at runtime. lets get this up-voted!!
LiLiu_Fabric
on:
Deployment Pipeline Run in Fabric monitor
on:
CI/CD support for Notebook Resources folder