Dynamic Batch Count in Data Factory Pipelines (FOREACH ACTIVITY)
Currently, in Data Factory, the ForEach activity requires a static batch count when executing in parallel mode. However, this fixed value does not always align with dynamic workloads, leading to inefficiencies. A more flexible approach would be to allow the batch count to be dynamic, adjusting automatically based on the number of items in the loop or available system resources.
For example, instead of manually setting a batch count of 10, users could define a dynamic value based on the total number of items, ensuring optimal parallel execution without overloading the system. This could be achieved by allowing expressions, pipeline parameters, or metadata-driven configurations to determine batch size at runtime. lets get this up-voted!!
1 Comment
- JannickVanGielRegular Visitor
Agreed. This is a must-have feature, which was already available in Azure Data Factory.
Recent ideas
Dedicated Fabric Message Center to communicate planned Fabric Changes
Summary Microsoft recently rolled out a change that automatically enabled the Item Recovery tenant setting in Microsoft Fabric for tenants where the setting had not been explicitly configured. Accor...Edgar_5 hours agoNew MemberNew23Views4likes0CommentsWorkspace-Level Control of Storage Modes (Import vs DirectQuery vs Direct Lake)
Organizations operating under strict governance and data residency requirements need the ability to control which storage modes are permitted within a workspace. Currently, report authors can choose...masoodaftab5 hours agoNew MemberNew4Views0likes0CommentsGranular Workspace Permissions Beyond Workspace Roles
a { text-decoration: none; color: #464feb; } tr th, tr td { border: 1px solid #e6e6e6; } tr th { background-color: #f5f5f5; } The current workspace access model is limited to the predefined workspac...masooddevcorp5 hours agoNew MemberNew13Views1like0CommentsAccessibility issue in the Share dialog of embedded Power BI reports
Accessibility issue in the Share dialog of embedded Power BI reports I would like to share an accessibility finding identified while testing an embedded Power BI report published as part of the cont...ugartema9 hours agoNew MemberNew10Views1like0Comments