Enable Table-Level Execution in Fabric Copy Job
a { text-decoration: none; color: #464feb; } tr th, tr td { border: 1px solid #e6e6e6; } tr th { background-color: #f5f5f5; } Fabric Copy Job currently executes the entire copy job when the Run button is selected. It would be very useful to have the ability to select one or more individual tables and execute only the selected tables without modifying the Copy Job configuration. For example, if a Copy Job contains 10 tables and the mapping or configuration of only one table is modified, we may want to immediately run only that table to validate and apply the change. Currently, running the Copy Job also processes the other tables even though they have not been changed. A simple implementation could be to add checkboxes next to each table in the monitoring/table list and provide an action such as "Run selected tables." This would be particularly useful for: Re-running a table after changing its mapping or configuration Re-loading only the tables that require correction Testing changes without unnecessarily processing unrelated tables Reducing unnecessary data movement and processing Ideally, the existing Run button would continue to execute the entire Copy Job, while a new Run selected tables option would execute only the tables selected by the user. This would allow us to keep multiple related tables in a single production Copy Job while still having the flexibility to execute individual tables when needed.
Show Refreshing indicator in workspace for semantic model refreshes from pipelines and XMLA
When I start a semantic model refresh by hand ("Refresh now") or from a schedule, the workspace list shows a spinning refresh icon next to the model. That way, anyone can see a refresh is running. That indicator doesn't show up when the refresh comes from: A Data Factory / Fabric pipeline (the Semantic model refresh activity) The XMLA endpoint (for example, SSMS, Tabular Editor, TMSL/TOM scripts, or Semantic Link) In these cases, the workspace list looks idle. The Status column is blank, and Refreshed still shows the last completed refresh. The only way to know a refresh is running is to open the model's Refresh history, which shows the run as In progress with type Data Factory.
Custom sort order for schemas in the Lakehouse explorer
Schemas in the Lakehouse explorer are always sorted alphabetically, so medallion layers appear as bronze, dbo, gold, silver. Please allow a custom display order, or pin schemas, so they can appear in a logical order (bronze → silver → gold) without renaming them with number prefixes, which require backticks in every query.
Run OnDemand Fabric Data Pipeline with Runtime Parameters using REST API
Currently, Microsoft Fabric Data Pipelines can be triggered on demand, but there is no native capability to pass runtime parameter values when initiating a pipeline execution through REST APIs. This limitation reduces flexibility for automation, orchestration, and operational support scenarios. FYR - Background Jobs - Run On Demand Execute - REST API (DataPipeline) | Microsoft Learn Problem Statement Many enterprise data integration workloads require dynamic values to be supplied at runtime, such as: Processing a specific business date Running selective backfills Environment-specific executions Customer- or region-specific processing Triggering pipelines from external applications CI/CD and DevOps automation workflows Without runtime parameter support, organizations must create workarounds such as configuration tables, temporary files, or multiple copies of the same pipeline, adding complexity and maintenance overhead. Proposed Enhancement Enhance Fabric Data Pipeline REST APIs to allow: Triggering a pipeline run with runtime parameter values. Overriding default pipeline parameters at execution time. Passing multiple parameter types: String Integer Boolean Decimal Date/DateTime Array/Object Viewing submitted parameter values in monitoring and run history. Invoking pipelines from external applications, Azure Functions, Logic Apps, Power Automate, and DevOps pipelines with custom parameter payloads. Please consider adding support for runtime parameter injection when triggering Fabric Data Pipelines through REST APIs. This capability is critical for enterprise automation and large-scale operational workloads.
Copy job for SAP with ABAP Add-On in Microsoft Fabric (Preview)
SAP systems sit at the center of many enterprises’ core business operations, powering processes across finance, supply chain, manufacturing, procurement, and HR. That makes SAP data some of the most business-critical data in the enterprise. As organizations modernize their analytics and AI platforms, bringing SAP data together with the rest of the enterprise data estate has become increasingly important. But moving SAP data at enterprise scale has historically been difficult. SAP landscapes are complex, data volumes are large, and extraction architectures often require specialized frameworks, custom code, and additional operational layers. For many organizations, this creates friction between where their most important operational data lives and where they want to analyze, enrich, and activate it. We are delivering the next step of our SAP roadmap: Copy job for SAP with ABAP Add-On in Microsoft, this new capability complements offerings like: SAP Business Data Cloud Connect was announced in SAP and Microsoft accelerate business insights and AI innovation with SAP Business Data Cloud Connect for Microsoft Fabric and will be available over the coming months). It will enable bi-directional zero-copy sharing between SAP Business Data Cloud and Microsoft Fabric without the need for data movement. Mirroring via SAP Datasphere, a turnkey data replication capability providing scalable near real-time data movement from SAP sources into Fabric OneLake (Generally Available). To learn more, refer to the Fabric mirroring documentation. Now, organizations can extract large volumes of SAP data through Copy job, reduce the need for external extraction frameworks, and build a scalable path from initial ingestion to ongoing incremental updates. Copy job provides a configuration driven experience for moving data across clouds, applications and on-premises systems – designed to support high-scale, multi-cloud data movement for petabyte-scale ingestion scenarios. Copy job helps bring data into OneLake as part of a unified, governed foundation for analytics and AI. High-performance SAP extraction with Copy job and Microsoft ABAP add-on integration This new integration option enables scalable, high-throughput extraction from SAP systems, making it easier than ever to bring large volumes of SAP data into Microsoft Fabric for analytics, reporting, and AI. Figure: High-level architecture diagram of Copy job with ABAP Add-on. Broad data coverage across SAP Systems Extracting data from SAP systems at scale can be complex and resource intensive. This new capability simplifies the process while delivering high performance. The solution runs directly inside SAP using a Microsoft-provided ABAP Add-On. Data is extracted at the application layer and transferred efficiently through Copy Job in Fabric Data Factory. This removes the need for complex external extraction frameworks and reduces operational overhead. It supports a wide range of SAP data sources. You can extract from SAP tables, views, and ABAP CDS views, including semantically rich models used for analytics. The capability works across both SAP ECC and SAP S/4HANA, whether deployed on-premises or in the cloud. Providing consistent access to both raw operational data and business-ready data models. Flexible data delivery styles for real-world SAP scenarios Through the integration with Copy Job, the solution handles data movement at scale. The integration is designed for high throughput and efficient processing of large datasets. Parallel execution capabilities help accelerate ingestion and reduce data transfer times. The solution supports different loading strategies depending on your needs. You can perform full snapshots for initial ingestion or bulk replication scenarios. For ongoing updates, incremental loads can be configured using watermark-based extraction. This allows you to process only the data that has changed. Together, these capabilities make it easier to design efficient pipelines. You can start with a large initial load and then transition seamlessly to incremental updates, without changing the overall architecture. SAP data in OneLake, ready for analytics and AI After ingestion, SAP data is immediately available in Microsoft OneLake. From there, it can be used across the entire Fabric platform. You can build Power BI reports directly on SAP data and operational datasets. You can run large-scale analytics workloads or combine SAP data with non-SAP sources from across your organization creating a unified data foundation without silos. With all data in one place, it also becomes easier to enable AI-driven scenarios. Applications and data agents can access consistent, trusted business data to generate insights and drive automation. Get started For more information, refer to the ABAP Add-On for SAP data extraction with Copy Job in Microsoft Fabric documentation. You can start exploring this capability today to: Accelerate SAP data ingestion Simplify data extraction architectures Enable scalable analytics and AI scenarios Looking ahead Copy job for SAP with Microsoft ABAP Add-On is another step in our continued investment in SAP data integration for Microsoft Fabric. SAP is one of the most important enterprise data sources for our customers, and Data Factory in Fabric is designed to help organizations bring SAP and non-SAP data together through scalable, flexible data movement experiences. With Copy job, customers can move data across clouds, applications, and on-premises systems into OneLake. With the Microsoft provided ABAP Add-On, that same Copy job experience now extends more deeply into SAP, helping customers bring business-critical SAP data into Fabric for analytics, reporting, and AI. We will continue expanding SAP data integration capabilities across Microsoft Fabric so customers can simplify their data architectures, reduce silos, and build a unified, governed foundation for enterprise analytics and AI.Playbook for metadata driven Lakehouse implementation in Microsoft Fabric
Co-Author – Gyani Sinha, Abhishek Narain Overview A well-architected lakehouse enables organizations to efficiently manage and process data for analytics, machine learning, and reporting. To achieve governance, scalability, operational excellence, and optimal performance, adopting a structured, metadata-driven approach is crucial for lakehouse implementation. Building on our previous blog, Demystifying Data Ingestion in Fabric, this post explores how to design and implement an end-to-end metadata-driven Lakehouse using Microsoft Fabric. While the core design principles are applicable to platforms like Azure Databricks and Microsoft Fabric, this blog focuses on leveraging Microsoft Fabric to create a scalable, governed, and high-performance lakehouse—rooted in real-world customer implementations. Basic components of metadata driven Lakehouse implementation framework Below graphic represents the basic building blocks (components) of the reference lakehouse implementation. Basic_building_blocks_of_metadata-driven_lakehouse_implementation Basic building blocks of metadata-driven Lakehouse implementations The table outlines the different components and their respective purposes. # Component name Purpose 1 Control Tables Implement a control table to orchestrate and manage data ingestion, validation, and processing in a Lakehouse. Control tables must store parameters and configurations for ingestion, validation, and ETL processes, enabling dynamic adjustments without modifying the code. 2 Data Ingestion Implement the metadata driven Data Ingestion component leveraging the control table to copy data from various source systems to the lakehouse. You may also use Shortcuts and the Mirroring feature in Microsoft Fabric to streamline ingestion and minimize data movement. Mirroring continuously replicates your existing data estate directly into Fabric's OneLake, including data from Azure SQL Database, Azure Cosmos DB, and Snowflake. 3 Data Validation Implement a data validation component to ensure the integrity of the migrated data. This component will help detect and resolve any discrepancies that might arise during the migration or ingestion process. By systematically validating the data, you can ensure its accuracy, completeness, and consistency, ultimately safeguarding the quality of the migrated datasets. 4 Data Profiling and Data Quality Implement data profiling to gain a comprehensive understanding of your data. This process provides a descriptive overview, highlighting aspects like missing values, datatype distributions, and key statistics. Additionally, it helps identify anomalies by triggering alerts, enabling proactive data quality management and ensuring the data is ready for further processing or analysis. 5 Transformation and enrichment Implement a data transformation and enrichment component to transform and enrich the data for consumption based on configurable transformation, enrichment, and standardization rules. 6 Auditing Implement auditing component to audit the records ingested and processed into lakehouse. Auditing includes identifying any errors or issues that may have occurred during the process, as well as reviewing performance metrics to identify areas for improvement. 7 Notification Implement notification component to notify in case of success or failure events. This is super critical to ensure that the operation team is notified during all critical events. 8 Config Management Implement centralized config management component for managing the configuration to different source and target systems. 9 Reporting Implement a reporting component to visualize metrics related to ingestion, validation, and transformation processes. This will provide the operations team with a single pane of glass, offering an end-to-end status of the lakehouse and enabling them to effectively monitor and manage all processes. Metadata driven framework implementation Now that we've covered the key components, let's explore how these elements can be integrated to create a robust, metadata-driven lakehouse implementation using Microsoft Fabric. Metadata-driven_lakehouse_implementation_using_Microsoft_Fabric Metadata-driven lakehouse implementation using Microsoft Fabric The diagram above highlights the key processing components, such as data ingestion, data profiling, and transformation, at the top, along with cross-cutting components like configuration management, auditing, notification, and reporting that support all processing activities. Key components used in building the metadata-driven lakehouse implementation include - Data ingestion – There are several approaches to implementing data ingestion in Fabric, including a low-code method using Microsoft Fabric Data Pipeline with pre-built source and destination connectors, or a code-based approach using Microsoft Fabric's Spark-based implementation. The destination in Fabric can vary based on the use case or customer preference, including lakehouse files, lakehouse tables, or Data Warehouse. Additionally, other ingestion methods like mirroring and shortcuts are available, which also need to be managed. The benefits of storing data in lakehouse files are it supports both structured and unstructured data in their original format, making it a reliable source of truth for future consumption. Metadata-driven_Data_Ingestion_component Metadata-driven Data Ingestion component The image illustrates the key modules and the high-level data ingestion flow, starting with the control module, which queries the 'ingest_control' table to manage and control the ingestion process based on the configured sources handled by the config module. The data ingestion module is responsible for transferring data into the target lakehouse and utilizes the notification module to send alerts in case of failures. All activities are logged in the 'ingest_audit' table and presented through the reporting module, offering a single pane of glass to monitor the ingestion process into the lakehouse. The monitoring module also tracks shortcut and mirrored items to ensure these are properly captured, using the 'shortcut_audit' and 'mirroring_audit' tables. Metadata-driven Ingestion reference implementation in Action Data validation – The validation module, typically written in Spark, uses the configuration to validate both the source data and the target lakehouse or data warehouse data, checking for completeness and reasonableness. The validation results are stored in the 'validation_results' table and are presented through the reporting module, providing the operations team with insights into any discrepancies introduced during the migration or ingestion process. Playbook_for_metadata_driven_Lakehouse_implementation_in_Microsoft_Fabric Data Validation Component The table provides examples of validation category, scope & criteria. Validation Category Validation Scope Validation Criteria Completeness Row Level Record Count Validation Completeness Column Level Columns Count Validation Completeness Column Level Columns Datatype Validation Reasonableness Column Level Numerical Columns Aggregation Check Reasonableness Column Level Categorical Columns Group By Count Validation Reasonableness Row Level Checksum Validation (first 100 rows) Data quality – The metadata-driven data quality component leverages metadata to automatically assess and enforce data quality rules across the lakehouse or data warehouse. It validates data consistency, accuracy, and compliance with predefined standards. The results are captured in a data quality results table and visualized through the reporting module, enabling the operations team to proactively identify and address any data quality issues. You can use Microsoft Purview’s Data Quality feature to scan lakehouse data and configure data quality rules for automated reporting of data health. Refer to Data Quality for Fabric Lakehouse in Unified Catalog for more information. PII anonymization - PII anonymization component involves identifying and anonymizing personally identifiable information (PII) from datasets to protect individuals' privacy while still allowing for data analysis and decision-making. This is essential in scenarios where sensitive data, such as names, addresses, and social security numbers, is involved. In a lakehouse scenario, PII anonymization is crucial for protecting data from unauthorized access and breaches, ensuring compliance with regulations like GDPR and CCPA, and maintaining data quality and integrity by preventing biases. This process helps organizations use data responsibly without compromising individuals' privacy. Transformation & enrichment – The transformation and enrichment component utilize the enrich control table to retrieve the list of datasets to be transformed, enriched, and stored from the bronze layer to the silver layer. This component handles both simple Spark-based transformations, such as renaming columns and changing data types, as well as more complex rule-based operations, like conditional filtering, any custom business logic or deriving new columns managed through 'transformation_config' table. Once the data is transformed, the serve control table is used to determine which datasets should be served into the gold layer after performing necessary aggregations. The audit module tracks the transformed records and logs the status of each run for accountability and traceability. Playbook_for_metadata_driven_Lakehouse_implementation_in_Microsoft_Fabric Transformation and Enrichment component Reference implementation in action. Bringing it all together using task flow in Microsoft Fabric End-To-End_Metadata-driven_Lakehouse_implementation_using_Microsoft_Fabric_in_ta End-To-End-Metadata driven Lakehouse implementation using Microsoft Fabric in task flow Conclusion Implementing a metadata-driven Lakehouse using Microsoft Fabric enables organizations to efficiently manage and process data with governance, scalability, and optimal performance. By following the structured framework outlined in this playbook, key components like data ingestion, validation, profiling, transformation, and auditing ensure data integrity and quality. This guide emphasizes the importance of cross-department collaboration for seamless implementation, unlocking the full potential of data to drive insights and innovation. We hope this playbook serves as a valuable resource for your Lakehouse implementation journey. Special shout out to Winnie and Dharmendra for their continued contributions!Fabric Workspace Operator Role for Pipeline Monitoring and Rerun Operations
Many organizations have dedicated production support and operations teams responsible for monitoring Microsoft Fabric pipelines and responding to failures. Currently, the available workspace roles are Admin, Member, Contributor, and Viewer. While Viewers can monitor content, they cannot rerun failed pipelines. Contributors can rerun pipelines, but they also have permissions to modify pipeline definitions and other workspace artifacts. This creates a governance gap for enterprises that require segregation of duties between development and operations teams. Proposed Solution Introduce a new built-in Operator workspace role with permissions such as: Allowed View pipeline definitions Monitor pipeline runs and run history View execution logs and error details Manually trigger pipeline execution Rerun failed pipeline runs Cancel running pipeline executions Access workspace monitoring capabilities Not Allowed Create, edit, or delete pipelines Create, edit, or delete Fabric artifacts Modify notebooks, lakehouses, warehouses, or dataflows Manage workspace access or permissions Change workspace settings Connect/disconnect Git repositories Business Value This role would: Support enterprise segregation of duties requirements. Enable production support teams to resolve operational issues without granting development permissions. Reduce the risk of accidental changes in production workspaces. Improve governance and compliance. Align Microsoft Fabric with operational models commonly used in enterprise data platforms. Many organizations need a role that sits between Viewer and Contributor, providing operational control without development privileges. An Operator role would significantly improve security, governance, and supportability for Fabric Data Factory workloads. Use Case: Production support team monitors Fabric pipelines, investigates failures, and reruns failed executions, but should not be able to modify pipeline definitions or other workspace assets.
