Dynamic Batch Count in Data Factory Pipelines (FOREACH ACTIVITY)
Currently, in Data Factory, the ForEach activity requires a static batch count when executing in parallel mode. However, this fixed value does not always align with dynamic workloads, leading to inefficiencies. A more flexible approach would be to allow the batch count to be dynamic, adjusting automatically based on the number of items in the loop or available system resources.
For example, instead of manually setting a batch count of 10, users could define a dynamic value based on the total number of items, ensuring optimal parallel execution without overloading the system. This could be achieved by allowing expressions, pipeline parameters, or metadata-driven configurations to determine batch size at runtime. lets get this up-voted!!
1 Comment
- JannickVanGielRegular Visitor
Agreed. This is a must-have feature, which was already available in Azure Data Factory.
Recent ideas
Power BI, import report and semantic model: show change connection pop-up
When you need to import an existing Power BI report and semantic model in Tenant B that's created in Tenant A, you need to jump through a number of hoops to fix the original connection. It's doable, ...Reitse46 minutes agoMost Valuable ProfessionalNew4Views0likes0CommentsEnable per-user security for shared gateway connections using service accounts
When a shared gateway connection uses a highly privileged service account, all source queries run with that account’s permissions. Users cannot see the stored credentials, but report developers who c...AshishChougala1 hour agoMicrosoft EmployeeNew6Views2likes0CommentsRequest to add a feature to extract gateway connection status and last time credentials used
Could you please add a feature in the admin UI on Microsoft Fabric/Power BI to extract the Gateway connections with details of the gateway connection status and last activity information along with t...Vinodkumar_Chak2 hours agoNew MemberNew21Views1like0CommentsPower BI - incorrect handling of empty string parameter in DAX query in paginated report
Blank string parameters in paginated reports are treated as empty strings in Power BI Report Builder but as null values in the Power BI Service, resulting in incorrect report output. -> Create a pag...roshinikumarkar5 hours agoMicrosoft EmployeeNew25Views5likes0CommentsAllow NotebookUtils getSecret() to authenticate with Workspace Identity
Current behavior In Microsoft Fabric, notebookutils.credentials.getSecret() authenticates against Azure Key Vault using the identity of the user who executes the notebook. This behavior appli...tmihara7 hours agoNew MemberNew20Views1like1Comment