Dynamic Batch Count in Data Factory Pipelines (FOREACH ACTIVITY)
Currently, in Data Factory, the ForEach activity requires a static batch count when executing in parallel mode. However, this fixed value does not always align with dynamic workloads, leading to inefficiencies. A more flexible approach would be to allow the batch count to be dynamic, adjusting automatically based on the number of items in the loop or available system resources.
For example, instead of manually setting a batch count of 10, users could define a dynamic value based on the total number of items, ensuring optimal parallel execution without overloading the system. This could be achieved by allowing expressions, pipeline parameters, or metadata-driven configurations to determine batch size at runtime. lets get this up-voted!!
1 Comment
- JannickVanGielRegular Visitor
Agreed. This is a must-have feature, which was already available in Azure Data Factory.
Recent ideas
Enable sorting of diagram pages in Power BI Desktop
I, as a developper, rely a lot on model view and diagrams... It's annoying to not be able to sort the diagram pages in this view, since I tend to try to organize these by subject and it becomes a...Daproulx8 minutes agoFrequent VisitorNew2Views0likes0CommentsImproving complexity of Page Navigator in PowerBI
The page navigator could be improved in multiple ways to help reports that have many pages: Allow scroll in vertical view Allow pages within the navigator (EX: using a right and left carrot to sho...kreid5 hours agoNew MemberNew77Views2likes2Comments