data factory
937 TopicsSupport parameterized file destination paths in Copy job
Currently, the file destination path in a Fabric Copy job is defined statically in the job configuration. Please add support for parameterizing the destination file path, including folder and file names, so that the output location can be dynamically controlled at runtime. For example, it should be possible to define an output path such as: Files/output/{run_date}/{run_id}/ where run_date and run_id are supplied dynamically at runtime. It should also be possible to pass these values from a pipeline or other orchestration process. This would enable a single Copy job definition to be reused across different executions without creating multiple Copy jobs that differ only in their destination paths. Use cases Create execution-specific output folders using run_date and run_id Dynamically partition output files by processing date Prevent outputs from different executions from being mixed or overwritten Pass destination path values from a pipeline or orchestration process Reuse the same Copy job definition across multiple executions and environments8Views0likes0CommentsREST API Should expose credentials used in connections
When a consultant leaves a client, it's important to clean up any connections that may have the consultants credentials embedded within. I'm able to use the Fabric CLI to get the managed connections for all items, but I can't see the user name used in them. It would be helpful to expose the user name (UPN) in the REST API and fab cli (and other SDKs, etc.). There are other cases where an admin or governing body would want to audit the credentials used in connections.34Views7likes0CommentsExpose full activity-level error details in Workspace Monitoring Eventhouse
Microsoft Fabric Workspace Monitoring exposes activity-level pipeline telemetry in FabricDataPipelinesActivityRunsLogs, including PipelineRunId, OperationId, ActivityIterationCount, ActivityName, ActivityType, Status and FailureType, however, it does not expose the full error information shown in the Fabric Monitoring Hub UI for the same failed activity. The Eventhouse log contains only a generic failure category such as UserError, while the UI can show the actual SQL, notebook, copy, dataflow or semantic-model error message, this prevents reliable centralized operational monitoring.. The Job Scheduler REST API exposes failureReason only at pipeline-run level but that message cannot be safely attributed to a single activity when a pipeline contains parallel branches or more than one failed activity. Please expose a supported, queryable activity-level diagnostics payload in Workspace Monitoring, either directly in the table FabricDataPipelinesActivityRunsLogs or in a documented related table/API. Requested fields: - ErrorCode - ErrorMessage - ErrorDetails or raw diagnostic payload - IsRetriable, where applicable - Correlation to PipelineRunId, OperationId and ActivityIterationCount The payload should be accessible through KQL and preserve the exact association with the failed activity instance, including retry and loop iterations. The Fabric Monitoring Hub already displays these details, so exposing them in the monitoring telemetry would allow organizations to build reliable central monitoring without adding custom OnFailure logging branches to every activity in every pipeline. Business impact: We operate a large Fabric Data Factory estate with existing and future pipelines. A central Eventhouse-based monitoring solution is required for operations and troubleshooting. Without activity-level error details, support teams must manually open each pipeline run in the Fabric UI. Custom per-activity failure handlers are not a scalable alternative because they materially increase the complexity and maintenance cost of every pipeline definition.6Views0likes0CommentsCopy job for SAP with ABAP Add-On in Microsoft Fabric (Preview)
SAP systems sit at the center of many enterprises’ core business operations, powering processes across finance, supply chain, manufacturing, procurement, and HR. That makes SAP data some of the most business-critical data in the enterprise. As organizations modernize their analytics and AI platforms, bringing SAP data together with the rest of the enterprise data estate has become increasingly important. But moving SAP data at enterprise scale has historically been difficult. SAP landscapes are complex, data volumes are large, and extraction architectures often require specialized frameworks, custom code, and additional operational layers. For many organizations, this creates friction between where their most important operational data lives and where they want to analyze, enrich, and activate it. We are delivering the next step of our SAP roadmap: Copy job for SAP with ABAP Add-On in Microsoft, this new capability complements offerings like: SAP Business Data Cloud Connect was announced in SAP and Microsoft accelerate business insights and AI innovation with SAP Business Data Cloud Connect for Microsoft Fabric and will be available over the coming months). It will enable bi-directional zero-copy sharing between SAP Business Data Cloud and Microsoft Fabric without the need for data movement. Mirroring via SAP Datasphere, a turnkey data replication capability providing scalable near real-time data movement from SAP sources into Fabric OneLake (Generally Available). To learn more, refer to the Fabric mirroring documentation. Now, organizations can extract large volumes of SAP data through Copy job, reduce the need for external extraction frameworks, and build a scalable path from initial ingestion to ongoing incremental updates. Copy job provides a configuration driven experience for moving data across clouds, applications and on-premises systems – designed to support high-scale, multi-cloud data movement for petabyte-scale ingestion scenarios. Copy job helps bring data into OneLake as part of a unified, governed foundation for analytics and AI. High-performance SAP extraction with Copy job and Microsoft ABAP add-on integration This new integration option enables scalable, high-throughput extraction from SAP systems, making it easier than ever to bring large volumes of SAP data into Microsoft Fabric for analytics, reporting, and AI. Figure: High-level architecture diagram of Copy job with ABAP Add-on. Broad data coverage across SAP Systems Extracting data from SAP systems at scale can be complex and resource intensive. This new capability simplifies the process while delivering high performance. The solution runs directly inside SAP using a Microsoft-provided ABAP Add-On. Data is extracted at the application layer and transferred efficiently through Copy Job in Fabric Data Factory. This removes the need for complex external extraction frameworks and reduces operational overhead. It supports a wide range of SAP data sources. You can extract from SAP tables, views, and ABAP CDS views, including semantically rich models used for analytics. The capability works across both SAP ECC and SAP S/4HANA, whether deployed on-premises or in the cloud. Providing consistent access to both raw operational data and business-ready data models. Flexible data delivery styles for real-world SAP scenarios Through the integration with Copy Job, the solution handles data movement at scale. The integration is designed for high throughput and efficient processing of large datasets. Parallel execution capabilities help accelerate ingestion and reduce data transfer times. The solution supports different loading strategies depending on your needs. You can perform full snapshots for initial ingestion or bulk replication scenarios. For ongoing updates, incremental loads can be configured using watermark-based extraction. This allows you to process only the data that has changed. Together, these capabilities make it easier to design efficient pipelines. You can start with a large initial load and then transition seamlessly to incremental updates, without changing the overall architecture. SAP data in OneLake, ready for analytics and AI After ingestion, SAP data is immediately available in Microsoft OneLake. From there, it can be used across the entire Fabric platform. You can build Power BI reports directly on SAP data and operational datasets. You can run large-scale analytics workloads or combine SAP data with non-SAP sources from across your organization creating a unified data foundation without silos. With all data in one place, it also becomes easier to enable AI-driven scenarios. Applications and data agents can access consistent, trusted business data to generate insights and drive automation. Get started For more information, refer to the ABAP Add-On for SAP data extraction with Copy Job in Microsoft Fabric documentation. You can start exploring this capability today to: Accelerate SAP data ingestion Simplify data extraction architectures Enable scalable analytics and AI scenarios Looking ahead Copy job for SAP with Microsoft ABAP Add-On is another step in our continued investment in SAP data integration for Microsoft Fabric. SAP is one of the most important enterprise data sources for our customers, and Data Factory in Fabric is designed to help organizations bring SAP and non-SAP data together through scalable, flexible data movement experiences. With Copy job, customers can move data across clouds, applications, and on-premises systems into OneLake. With the Microsoft provided ABAP Add-On, that same Copy job experience now extends more deeply into SAP, helping customers bring business-critical SAP data into Fabric for analytics, reporting, and AI. We will continue expanding SAP data integration capabilities across Microsoft Fabric so customers can simplify their data architectures, reduce silos, and build a unified, governed foundation for enterprise analytics and AI.3.7KViews1like4CommentsOn-premises data gateway August 2026 release
The August 2026 release of the on-premises data gateway is version 3000.330. The new August 2026 release of the on-premises data gateway (version 3000.330) continues our commitment to delivering secure, reliable, and manageable connectivity between on-premises data sources and Microsoft Fabric services. This release includes improvements focused on gateway manageability, operational visibility, enterprise governance, and platform reliability, helping organizations manage their gateway environments with greater confidence. It also brings compatibility updates for the August 2026 release of Power BI Desktop, ensuring consistent query execution and a seamless experience between Power BI Desktop and cloud-based refresh scenarios. Additionally, this release includes enhancements across security, authentication, diagnostics, and overall platform quality. We have received a few questions from community members regarding CVEs identified in third-party components used by the on-premises data gateway. As part of our ongoing commitment to security and reliability, we continuously evaluate and update third-party dependencies. These updates are incorporated into gateway releases following comprehensive security reviews, compatibility validation, and product qualification testing to help ensure a secure, stable, and reliable experience for customers. Security and dependency updates are delivered through regular gateway releases. We recommend keeping your gateway deployment up to date to benefit from the latest security enhancements, dependency updates, performance improvements, and platform reliability fixes. Power BI Desktop compatibility This update brings the on-premises data gateway up to date with the August 2026 release of Power BI Desktop. Download on-premises data gateway (standard mode) Download on-premises data gateway (personal mode) This version of the gateway will ensure that the reports that you publish to the Power BI Service and refresh via the gateway will go through the same query execution logic/run-time as in the August version of Power BI Desktop. Next steps Upgrade to version 3000.330 to take advantage of the latest security, authentication, and diagnostics improvements. We encourage you to share feedback and feature requests through the Power BI Ideas forum to help shape future gateway investments.2.7KViews0likes6CommentsParameterize Dataflow Gen2 data source connection
Allow the Dataflow Gen2 data source connection GUID to be parameterized and stored in a Variable Library. This would allow the same Dataflow to use different connections depending on the workspace/environment, without modifying the Dataflow itself. Hypothetical example: Variable Library: SourceConnectionGuid = 01234567-89ab-cdef-0123-456789abcdef The Dataflow references this variable for its data source connection. In another workspace, the same Dataflow could use: Variable Library: SourceConnectionGuid = fedcba98-7654-3210-fedc-ba9876543210 This would make it much easier to use the same Dataflow across personal feature workspaces, test, and production while keeping the appropriate connection for each environment.45Views2likes1CommentMake Prep Data for AI Deployable Across Fabric Environments
Idea Fabric Deployment Pipelines should consistently promote all Prep Data for AI settings with semantic models, including: AI Instructions Verified Answers AI schema and field selections Today, deploying from Dev to UAT or Production can remove, reset, or inconsistently transfer these configurations. Git/TMDL updates may also cause AI Instructions to return blank. Why this matters Without reliable deployment, companies must manually recreate and validate AI configurations in every environment. This: Breaks standard CI/CD processes Creates inconsistent Copilot responses Introduces governance and production risks Prevents enterprise-scale adoption Requested improvements Full Prep Data for AI support in Deployment Pipelines REST APIs to read and write the configuration Reliable Git and TMDL integration Deployment validation for missing or changed AI settings Microsoft Support has indicated that the feature is still in Preview, so support and workarounds are currently limited. Enterprise customers need a scalable, auditable deployment process—not manual configuration in each workspace. If your organization uses separate Dev, Test, and Production environments, please vote. This capability is essential for making Fabric Copilot enterprise-ready.59Views4likes0CommentsProgrammatic point-of-failure recovery for Fabric pipelines
The Fabric monitoring UI already supports Rerun → rerun from failed activity. That capability is only reachable by a human clicking in the portal. Please make point-of-failure recovery available to automation as well through whatever mechanism fits the platform best. When a long pipeline fails at a late step, anything automated can only restart it from the beginning. Re-running from step one re-executes hours of already-successful work, burns capacity, delays the SLA, and risks duplicate loads or re-triggered downstream refreshes where activities aren't perfectly idempotent. So today recovery either requires someone to open the portal just to make reruns resumable. A supported way to resume a failed pipeline run from its point of failure without a person in the portal. I'm deliberately not prescribing the mechanism. Any of these would solve it: A recovery option on the existing job scheduler API A "rerun from failure" action available to Activator rules on pipeline failure events A pipeline-level setting for automatic resume-on-retry A first-class checkpoint/resume capability that survives a plain re-run Something else the product team considers a better fit Thanks for your attention,23Views2likes0CommentsWorkspace Outbound Access Protection (OAP) for Operations Agent and Fabric Maps (Preview)
Co-authors: Andre Terceros and Bodhisatva Gautam Workspace Outbound Access Protection (OAP) in Microsoft Fabric helps WS admins secure outbound connections from workspace items to external resources. Administrators can control outbound access by blocking unwanted connections by default and allowing only approved connections through configured rules. As organizations adopt AI-powered operations at scale, governance and security remain at critical requirements. With this preview release, Microsoft Fabric introduces Outbound Access Protection (OAP) for Operations Agent and Fabric Maps, enabling workspace administrators to control the outbound actions an agent can perform. OAP for Operations Agent When OAP is enabled, Operations Agent continues to perform core functions including reasoning, recommendation generation, rule evaluation, and telemetry collection. However, outbound actions are governed by the workspace's configured access policies. Administrators gain greater visibility through in-product notifications, Teams messaging experiences, and the Operations Agent Activity Log, making it easier to identify and troubleshoot blocked actions. Figure: Outbound Access Protection helps workspace administrators govern outbound Operations Agent actions while providing clear visibility when actions are blocked. What's new with Operations Agent and OAP? With Outbound Access Protection support for Operations Agent, workspace administrators gain more control over how agent-initiated actions interact with external services and resources. The following updates improve governance, visibility, and operational oversight while helping organizations continue to automate with confidence. Govern outbound agent actions through workspace-level OAP policies. Control whether Operations Agent can send Teams notifications based on allowed connections. Prevent unauthorized cross-workspace actions when OAP policies restrict outbound access. Receive clear visibility when actions are blocked through in-product notifications and Teams messaging experiences. Monitor agent activity and OAP-related outcomes through the Operations Agent Activity Log. Figure: Activity Log Operations Details show the steps the agent completed and each action’s status, including when an action is blocked. OAP for Maps Fabric Maps can connect to a variety of data sources, including Lakehouse across Fabric workspaces, Kusto databases (KQL), Ontologies as well as external geospatial services such as Web Map Services (WMS), Web Map Tile Services (WMTS), and Web Feature Services (WFS). Workspace Outbound Access Protection helps organizations maintain security and compliance by governing outbound connections and requiring explicit approval before a map can access external resources. With OAP enabled, Fabric Maps follows a "default deny" model, ensuring only approved destinations can be accessed. How it Works When Workspace Outbound Access Protection is enabled on a workspace, Fabric Maps evaluates outbound connectivity at multiple stages: Save and Load Operations Whenever a map is created, updated, or loaded, Fabric evaluates all referenced data sources against the workspace policy. If a data source isn't permitted: References are blocked Disallowed sources are redacted when the map is opened Attempts to save unsupported references are rejected Runtime Data Access Map visuals continuously retrieve data such as: Map tiles Geospatial features Query results Before each outbound request is executed, Fabric validates the destination against the workspace's outbound access protection policy. External Service Validation External geospatial services such as WMS, WMTS, and WFS are validated through Data Movement and Transformation Services (DMTS). DMTS acts as a policy enforcement layer and ensures only explicitly approved external endpoints can be reached. Supported Connection Scenarios Lakehouse Lakehouse connectivity receives the most flexibility under the current release. Lakehouse within the same workspace are always allowed. Lakehouse in other workspaces can be allowed using Data Connection Rules. Administrators maintain granular control over cross-workspace access. External Geospatial Services Organizations can securely connect to approved WMS, WMTS, and WFS services. By using Data Connection Rules with the Geospatial Web Services connection type, administrators can create an allow list of approved endpoints. This enables scenarios such as: Accessing enterprise GIS systems Consuming approved mapping services Integrating with external geospatial data providers Kusto Databases and Ontologies Connections to Kusto databases and Ontologies within the same workspace continue to function normally. Cross-workspace connectivity for these sources is currently blocked when OAP is enabled. Future releases will introduce additional policy controls for these connection types. Configuring Fabric Maps with Workspace OAP Enabling protection is straightforward: Enable Workspace Outbound Access Protection for the workspace. Create Data Connection Rules that define approved destinations. Add approved external geospatial services using the Fabric connection experience. Configure Geospatial Web Services rules to authorize required endpoints. Once configured, Fabric Maps can access only the approved destinations specified by policy. Security by Design Workspace OAP for Fabric Maps follows several key security principles: Explicit Allow Lists: Only approved destinations can be reached. All other outbound connectivity is automatically blocked. Consistent Enforcement: Policies are applied during authoring, loading, and runtime execution, ensuring continuous protection throughout the lifecycle of a map. Fail-Closed Protection: If policy validation services become unavailable, Fabric denies cross-workspace connections by default. This fail-closed behavior helps prevent unintended data exposure during service disruptions. What’s next? We are actively working to expand OAP support for additional experiences and plan to add support for Power BI Semantic Models and Reports soon in GA soon. Your feedback is essential! Let us know how we can make Fabric even more secure and flexible for your workloads by sharing your feedback at Fabric Ideas – Microsoft Fabric Community Learn more Workspace outbound access protection overview Outbound Access Protection for Operations Agent (Preview) Workspace Outbound Access Protection for Fabric Maps346Views0likes0Comments