pipelines
916 TopicsBuild more connected workflows with Fabric Actions and Business Actions in Fabric Pipelines (Preview)
Modern data workflows rarely stop at moving and transforming data. Once data is ready, you may need to trigger another operation in Fabric, refresh a semantic model, update a business application, send a notification, generate a document, or kick off the next step in a business process. With Fabric Actions and Business Actions activities in Microsoft Fabric Data Factory, you can bring these steps directly into your Fabric pipelines. Together, these activities make it easier to orchestrate workflows across Fabric and the business applications your organization relies on, without having to build and maintain custom API integrations for every scenario. We’re continuing to expand what you can do with these activities, with support for more Fabric operations and 25 additional third-party connectors for Business Actions. Bring Fabric operations directly into your pipelines The Fabric Actions activity lets you invoke supported Fabric REST API operations directly from a pipeline. Instead of manually constructing REST API requests and managing the underlying authentication and request details, you can select the Fabric operation you need, configure its inputs, and incorporate it into the rest of your pipeline workflow. We’re expanding the range of supported Fabric actions across experiences such as Lakehouse, Warehouse, Notebook, Copy Job, Dataflow, Eventhouse, Eventstream, semantic models, OneLake shortcuts, Workspaces, Capacity, and more. This gives you more ways to automate Fabric as part of an end-to-end workflow. For example, you could run a Notebook as part of your data processing flow, manage a OneLake shortcut, perform a Lakehouse operation, or refresh a semantic model after new data is ready, all orchestrated from the same pipeline. Fabric Actions also provides built-in connection and credential handling, helping reduce the authentication and integration work that would otherwise be required when calling supported Fabric APIs directly. The following visual shows how you can select and configure a Fabric operation directly within your pipeline. Connect your pipelines to more business applications While Fabric Actions helps you orchestrate operations within Fabric, Business Actions helps connect your data workflows to the business applications and services where business processes happen. With this release, we’re expanding Business Actions with 25 additional third-party connectors, opening new possibilities across CRM, communications, document generation and signing, security, financial planning, project management, marketing, and more. For example, you can use Business Actions to: Send SMS notifications with D7SMS. Generate contracts, invoices, proposals, and other documents with Formstack Documents. Automate CRM and sales processes with connectors such as Pipeliner CRM and Commercient. Incorporate electronic signature workflows with Nitro Sign Enterprise Verified. Connect pipeline workflows with project and portfolio management systems. Integrate security and threat intelligence operations with services such as RiskIQ. Automate website content publishing with WP Connectr for WordPress. These additions build on the Business Actions experience to give pipeline authors more ways to connect data with the systems and processes that teams use every day. The following visual highlights some of the business applications that can be connected to your pipeline using Business Actions. Find and configure the action you need As we expand the actions available to pipeline authors, we’re also making them easier to discover and configure. Discover Fabric operations With the updated Fabric Actions experience, you can browse or search for supported Fabric operations instead of needing to know the underlying REST API endpoint ahead of time. Once you select an operation, the activity surfaces the relevant configuration and required parameters. You can then select an existing Fabric connection or create one, provide the required inputs, and use dynamic content where supported. The following visual shows the updated experience for discovering the Fabric operation you want to use. Configure your action Getting started is straightforward. Add Fabric Actions or Business Actions to your pipeline, find the operation you want to perform, configure the required inputs and connection, and incorporate the activity into your pipeline flow. For Fabric Actions: Create or open a pipeline in Microsoft Fabric Data Factory. Add the Fabric Actions activity to the pipeline. Browse or search for the Fabric operation you want to perform. Select or create a Fabric connection. Configure the required inputs. Save and run your pipeline. When the activity runs, the selected Fabric operation executes as part of the pipeline, with its response available through the activity output. Business Actions follows the same principle by providing structured configuration for supported business application operations, so you can incorporate business processes without starting from a custom API integration. Go from data workflows to business workflows The real power comes from combining Fabric Actions and Business Actions with the rest of your pipeline. Consider a pipeline that processes the latest sales data. Once processing completes, the same pipeline could refresh the semantic model used for reporting, create or update a record in a CRM system, generate a customer document, and send a notification that the latest information is ready. Or imagine a document workflow where a pipeline processes business data, generates a contract from that data, starts an electronic signature process, and continues downstream processing once the appropriate business step has taken place. Rather than building separate integrations for each part of the process, you can bring data processing, Fabric operations, and business actions together as part of a larger orchestrated workflow. This allows your pipelines to coordinate not only what happens to your data, but what happens next because of your data. The following visual shows an example of an end-to-end pipeline that brings data processing, Fabric operations, and business processes together. More actions, more possibilities We’re continuing to expand both Fabric Actions and Business Actions to give pipeline authors more ways to connect their data workflows with the next step in their business process. For Fabric Actions, that means broader coverage across Fabric workloads and platform operations, from Lakehouse, Warehouse, Notebook, and Copy Job to semantic models, OneLake shortcuts, Workspaces, Capacity, and more. For Business Actions, that means expanding the connector ecosystem so pipelines can interact with more of the applications and services organizations use every day. Our goal is to make these integrations easier to discover, configure, and incorporate into the pipelines you already build, so you can spend less time building integration plumbing and more time building end-to-end workflows. Get started To get started, open a pipeline in Microsoft Fabric Data Factory and explore the Fabric Actions and Business Actions activities. Whether you’re automating an operation within Fabric, connecting your pipeline to a downstream business application, or building a workflow that spans both, Fabric Actions and Business Actions give you more ways to turn data into action directly from Fabric Pipelines. We’re excited to continue expanding the actions you can connect and automate as part of your Fabric data workflows. To learn more, check out the Use the Fabric actions activity in a pipeline and Use the business action activity in a pipeline documentation.44Views0likes0CommentsWill Invoke Pipeline (Legacy) be deprecated?
Is there currently any plan or timeline to deprecate Invoke Pipeline (Legacy) in Microsoft Fabric? For same-workspace pipeline orchestration, the Legacy activity is still very convenient. The new Invoke Pipeline activity provides additional capabilities, such as cross-workspace invocation and child-pipeline monitoring, but it also requires a Connection object. Given the performance concerns that have been reported by multiple users, I’m wondering what the recommended approach is: Is it safe to continue using Invoke Pipeline (Legacy) for now? Does Microsoft recommend migrating existing pipelines to the new Invoke Pipeline activity? Is there any indication that the Legacy activity will eventually be deprecated or removed? Should we already start rebuilding existing pipeline orchestration using the new activity and Connections? It would be great to understand the intended direction, especially before investing time in rebuilding existing pipeline implementations. Thanks!52Views0likes7CommentsHow can AI agents improve Microsoft Fabric Data Factory pipeline automation?
I am exploring how AI agents can help make data pipeline development and monitoring more efficient in Microsoft Fabric Data Factory. Some potential use cases: Automatically creating pipeline workflows based on business requirements Detecting pipeline failures and suggesting fixes Optimizing data movement and transformation steps Monitoring data quality and identifying anomalies Triggering automated actions through APIs or external tools I would like to understand how teams are currently combining AI agents with Fabric Data Factory. Are you using: AI-assisted pipeline generation? Automated error analysis and recovery workflows? Fabric notebooks with AI models for data operations? Custom agents connected with Data Factory pipelines? Would appreciate insights, real-world examples, or architecture patterns from developers working with Fabric Data Factory automation.16Views0likes1CommentWhat Are the Best Practices for Building Reliable and Scalable Data Pipelines?
As organizations integrate data from multiple applications, databases, APIs, and cloud platforms, building reliable data pipelines becomes increasingly important. A pipeline that works well with small datasets may encounter problems as data volumes grow and workflows become more complex. Failed activities, dependency issues, schema changes, and inconsistent execution can affect downstream processes and reporting. I'd like to learn how the community handles these challenges in Microsoft Fabric Data Factory. A few questions: How do you design pipelines to handle large datasets efficiently? What is your approach to implementing retries, failure handling, and recovery mechanisms? How do you manage dependencies between multiple pipelines and activities? What strategies do you use for incremental data loading and avoiding duplicate records? How do you monitor pipeline execution and identify failures before they affect downstream systems? I'd especially appreciate practical examples involving Microsoft Fabric Data Factory, pipeline orchestration, and automated monitoring. What practices have helped you build more reliable data pipelines?14Views0likes1CommentFaulting application FabricPipelineWorker.exe
Hello, we installed an On-premise data gateway version 3000.330.1 (August 2026) on a Windows Server 2025. Every time we connect to SAP over SAP Table Application Server to our SAP System on SAP Cloud, we get the following error: Application: FabricPipelineWorker.exe Framework Version: v4.0.30319 Description: The process was terminated due to an unhandled exception. Exception Info: System.InvalidOperationException at Microsoft.DataTransfer.Common.ProcessManagement.Communication.MessageHandlerSet.Dispatch(Microsoft.DataTransfer.Common.ProcessManagement.Communication.IMessageChannel, Microsoft.DataTransfer.Common.ProcessManagement.Communication.Message) at Microsoft.DataTransfer.Common.ProcessManagement.Communication.MultiplexingMessenger.OnMessageToOtherChannel(Microsoft.DataTransfer.Common.ProcessManagement.Communication.IMessageChannel, MessageToOtherChannel) at Microsoft.DataTransfer.Common.ProcessManagement.Communication.MessageHandlerSet.Dispatch(Microsoft.DataTransfer.Common.ProcessManagement.Communication.IMessageChannel, Microsoft.DataTransfer.Common.ProcessManagement.Communication.Message) at Microsoft.DataTransfer.Common.ProcessManagement.Communication.IMessengerExtensions+<WaitForAsync>d__0`1[[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].MoveNext() at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task) at Microsoft.DataTransfer.Common.ProcessManagement.Communication.IMessengerExtensions+<WaitForAsync>d__1`1[[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].MoveNext() at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task) at Microsoft.DataTransfer.Common.ProcessManagement.RemoteService.RemoteServiceEnvironment+<DisposeServiceStubsAsync>d__4.MoveNext() at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task) at Microsoft.DataTransfer.DataMovementGateway.WorkerProcess.WorkerContainer+<StartAsync>d__11.MoveNext() at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task) at Microsoft.DataTransfer.DataMovementGateway.WorkerProcess.Program+<Main>d__1.MoveNext() at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task) at Microsoft.DataTransfer.DataMovementGateway.WorkerProcess.Program.<Main>(System.String[]) AND: Faulting application name: FabricPipelineWorker.exe, version: 5.68.9706.2, time stamp: 0x900a871d Faulting module name: KERNELBASE.dll, version: 10.0.26100.33158, time stamp: 0x4332201f Exception code: 0xe0434352 Fault offset: 0x00000000000c75fa Faulting process id: 0x2234 Faulting application start time: 0x1DD3B1901C52B57 Faulting application path: C:\Program Files\On-premises data gateway\FabricIntegrationRuntime\5.0\Shared\FabricPipelineWorker.exe Faulting module path: C:\WINDOWS\System32\KERNELBASE.dll Report Id: a3bc643f-227b-4f9e-b7b7-45ff005eb05f Faulting package full name: Faulting package-relative application ID: Has anyone an idea? BR ThomasSolved239Views1like10CommentsInvoke Pipeline "Rerun from Failed" re-executes entire child pipeline instead of skipping succeeded
Description: I have a parent-child pipeline setup in Fabric Data Factory, where the parent pipeline calls a child pipeline using the Invoke Pipeline activity. When one activity inside the child pipeline fails and the rest succeed, using "Rerun from failed" on the parent pipeline causes the entire child pipeline to re-execute from the beginning — including activities that already succeeded in the previous run. This is different from Azure Data Factory, where rerunning a parent pipeline from a failed activity correctly propagates the skip logic into nested Execute Pipeline calls — only the activities that actually failed (and anything downstream of them) get re-executed inside the child, while previously succeeded activities are marked Skipped. Steps to reproduce: Create a parent pipeline with an Invoke Pipeline activity calling a child pipeline. In the child pipeline, have multiple sequential/parallel activities where one fails and the rest succeed. Let the parent pipeline fail (due to the child failure). Use "Rerun from failed" on the parent pipeline run. Drill into the child pipeline's rerun — observe all activities (including previously succeeded ones) show fresh execution timestamps rather than being skipped. Impact: Non-idempotent activities (e.g., inserts without preceding deletes, counters, notifications, API calls) risk running twice, causing duplicate data or side effects. Increases pipeline runtime and cost unnecessarily on every rerun, since successful long-running steps are always redone. Breaks the expected "resume from point of failure" behavior that exists in ADF, making migration from ADF to Fabric riskier for complex orchestration patterns. Question for the community/Microsoft team: Is this a known/expected limitation of the current Invoke Pipeline implementation, or a bug? Is there a roadmap item to bring Fabric's rerun-from-failed behavior to parity with ADF for nested pipeline scenarios? In the meantime, is there a supported way to resume only the failed child pipeline run in a way that's reflected back in the parent's run status?174Views1like2CommentsMaterialized Lake View Refresh activity Pipeline bug
Hi all, When using the preview feature refresh mlv activity in the pipeline, is do have some major bugs. After saving, it goes back to default, the pipeline does not want to refresh after, and I cannot parametrize it. Is there any workarround, and is this a known bug? Cause this is not workable.353Views0likes10Comments[BUG] Git export downgrades SharePoint Online File (Preview) connector and pipeline corrupts
When using the new SharePoint Online File (Preview) connector in a GetMetadata activity inside a Fabric Data Factory pipeline, the Fabric workspace Git integration incorrectly downgrades the connector schema during export to Azure DevOps, and the pipeline gets corrupt when deploying to another workspace Environment: Microsoft Fabric Data Factory Git integration with Azure DevOps Deployment via fabric-cicd 1.0.0 Connection type: SharePoint Online File (Preview) Activity type: GetMetadata + Notebook in the same pipeline Steps to reproduce: Create a DataPipeline in Fabric Data Factory containing two activities: a GetMetadata activity using SharePoint Online File (Preview) as the connection type with Excel format, and a Notebook activity. Open the pipeline's JSON code view directly in the Fabric UI — the location type is correctly serialized as: json "location": { "type": "SharePointOnlineFileStoreLocation" }, "storeSettings": { "type": "SharePointOnlineFileStoreReadSettings" } Commit the pipeline to Azure DevOps via the workspace Git integration. Inspect the exported pipeline-content.json in the Azure DevOps repository — the location type has been silently downgraded to: json "location": { "type": "DatasetLocation" }, "storeSettings": { "type": "StoreReadSettings" } Deploy the pipeline from the repository to a production workspace using fabric-cicd — the pipeline arrives corrupted with error: Could not load resource 'pipeline_name'. Please ensure no mistakes in the JSON and that referenced resources exist. Status: UnknownError, Possible reason: Additional observation (key to isolating the bug): The downgrade only occurs when both a GetMetadata (SharePoint Online File Preview) activity and a Notebook activity are present in the same pipeline, if I disable the Notebook activity and commit the changes to DevOps it works926Views1like7CommentsCan parallel pipeline runs upsert into the same Lakehouse table without commit conflicts?
I want to ingest from 10 DB2 databases into a Lakehouse using one generic pipeline, invoked once per DB. They all write into the same Bronze tables (e.g. .raw_<table>). Every row carries a _db_id, so no two runs ever touch the same record. The Copy activity uses Table action = Upsert with key columns _db_id + the source primary key. The tables are not partitioned or clustered. In DESCRIBE HISTORY the commits show operation = MERGE, engineInfo = datafactorycopy, and operationParameters, operationMetrics, readVersion and isBlindAppend are all NULL. I would like to run the DBs in parallel, but we are unsure about commit conflicts. The concurrency control article says conflicts depend on the files a MERGE reads, and that the partition column must be a literal in the merge condition. It describes Spark, though, and I could not find anything about the Copy sink. Questions: Does the Copy activity upsert read the whole target table during the MERGE, or can it prune files, for example by _db_id? Can two parallel Copy upserts (different DBs, same table) fail with something like ConcurrentAppendException? Does the sink retry internally? Would partitioning the table by _db_id help with the Copy sink, even though we cannot set a literal predicate? Is there a recommended pattern for this, for example append into per-DB staging tables and a single MERGE afterwards? For now we run the DBs sequentially. Any experience or documentation pointers would be appreciated.Solved78Views0likes5CommentsError handling pattern for ForEach with partial failures: continue and report, or fail fast?
Hi everyone, I have a pipeline that loops over a list of tables with ForEach and runs a Copy activity per item. When one table fails, I would like the remaining tables to still load, and to get one consolidated failure report at the end instead of the whole run failing on the first error. The pattern I am considering: Wrap the Copy activity so its failure path (the On fail dependency) logs the table name and error message to a control table, and the ForEach item itself finishes as succeeded. After the loop, a Script or Lookup activity counts rows in the control table for this run, and a Fail activity raises one error with the summary if the count is above zero. A sketch of the final check: SELECT COUNT(*) AS failed_tables FROM ctl.load_log WHERE run_id = @{pipeline().RunId} AND status = 'Failed'; Questions: Is this the usual approach, or is there a built-in way to get partial-success semantics for a ForEach? How do you make sure the pipeline run status still reflects the failures so alerts fire? Any gotchas with the sequential versus parallel setting when logging from inside the loop? Thanks in advance.Solved39Views0likes3Comments