Dynamic Batch Count in Data Factory Pipelines (FOREACH ACTIVITY)
Currently, in Data Factory, the ForEach activity requires a static batch count when executing in parallel mode. However, this fixed value does not always align with dynamic workloads, leading to inefficiencies. A more flexible approach would be to allow the batch count to be dynamic, adjusting automatically based on the number of items in the loop or available system resources.
For example, instead of manually setting a batch count of 10, users could define a dynamic value based on the total number of items, ensuring optimal parallel execution without overloading the system. This could be achieved by allowing expressions, pipeline parameters, or metadata-driven configurations to determine batch size at runtime. lets get this up-voted!!
1 Comment
- JannickVanGielRegular Visitor
Agreed. This is a must-have feature, which was already available in Azure Data Factory.
Recent ideas
Power BI - Org apps and cross-report drillthrough
This idea is a request to provide cross-report drill-through capability for org apps. This means being able to navigate between reports hosted in different org apps. The following sections detail why...TomMartens4 hours agoSuper UserNew1.4KViews28likes4CommentsLet Workspace Admins manage Fabric Policies
For example, allow a Workspace Admin to grant other identities the Contributor role, while also granting them higher privileges in certain areas, e.g. List Workspace role permissions (without resha...frithjof_v12 hours agoCommunity ChampionNew35Views0likes2CommentsEnforcing Fabric Items naming standard policy
A standardized naming framework through centralized management for Fabric items similar to one in SQL Server Policy Based Management that enforces naming standards and naming conventions compliance d...ayamas13 hours agoRegular VisitorNew106Views0likes1CommentRun OnDemand Fabric Data Pipeline with Runtime Parameters using REST API
Currently, Microsoft Fabric Data Pipelines can be triggered on demand, but there is no native capability to pass runtime parameter values when initiating a pipeline execution through REST APIs. This ...singhs814 hours agoNew MemberNew43Views2likes2Comments