Monitor your Eventstreams with Workspace Monitoring — Now with per-Eventstream control (Preview)
When a real-time data pipeline breaks, the first question is always the same: what went wrong, and where? Eventstream observability in Microsoft Fabric brings that visibility directly into Workspace Monitoring, so you can track the health, performance, and errors of your eventstreams without leaving the Fabric portal or setting up external monitoring infrastructure. With this update, eventstream observability returns with a redesigned approach that puts you in control — including a per-eventstream toggle that lets you decide exactly which eventstreams emit monitoring data and which don't. Why observability matters for real-time pipelines Real-time data pipelines operate continuously. Unlike batch jobs that run and finish, an eventstream can process data for days or weeks without anyone checking on it. When something goes wrong — a destination write error, or backlogged events — the problem often goes unnoticed until downstream consumers report stale or missing data. Eventstream observability closes that gap. When enabled, each eventstream emits structured metrics and error data to your workspace's monitoring Eventhouse. This data lands in three dedicated KQL tables that you can query, alert on, and build dashboards against: Table What it captures Emission frequency When to use it EventStreamMetrics Throughput, event counts, ingress and egress rates ~1 minute Capacity planning, data flow validation, throughput monitoring EventStreamErrorMetrics Error category, error code, severity, affected events ~1 minute Troubleshooting failures, identifying recurring error patterns EventStreamNodeStatus Node state (running, stopped) ~6 hours Health checks, topology status overview Once monitoring is enabled for an eventstream, these tables appear in your workspace's monitoring KQL database alongside existing Eventhouse and other workload tables. Figure: Eventstream monitoring tables in the Workspace Monitoring KQL database, showing EventStreamMetrics, EventStreamErrorMetrics, and EventStreamNodeStatus tables available for querying. What changed: Per-eventstream control During the initial preview, eventstream observability was enabled globally for all eventstreams in a workspace. Based on customer feedback, the feature now provides granular control — you choose exactly which eventstreams emit monitoring data, giving you flexibility to focus observability on the pipelines that matter most. The key changes: Per-eventstream toggle. A new setting — Log Eventstream activity — appears in each eventstream's settings panel. You can enable or disable observability for each eventstream individually, so a workspace with ten eventstreams can monitor only the two or three that are most critical. Default OFF. All eventstreams default to monitoring OFF. You choose which ones to enable. How to get started Eventstream observability requires Workspace Monitoring to be enabled on your workspace. If you haven't set it up yet: Open your workspace settings. Navigate to the Monitoring section. Enable Workspace Monitoring and wait for the monitoring Eventhouse to provision. Once Workspace Monitoring is active: Open any eventstream in your workspace. Select the Settings gear icon. Find the Log Eventstream activity toggle. Enable it. Within a few minutes, data begins flowing into the eventstream tables in your monitoring KQL database. You can query them directly with KQL, build Real-Time Dashboards, or use KQL querysets for deeper analysis. What comes next This release covers performance metrics, error metrics, and node health — the foundation for operational observability. Additional capabilities are planned for future updates: Diagnostic logs — Error logs from the ASA processing engine, providing deeper root-cause analysis for failures like deserialization errors, runtime query errors, and output write failures. Expanded connector coverage — Monitoring support for connector-based sources is under investigation and will be added in a future phase. Next steps Refer to the Eventstream monitoring in Workspace Monitoring documentation to learn more about configuration and supported tables. Refer to the Workspace Monitoring overview documentation for setup instructions. Share your feedback and feature requests in the Fabric Community Forums. Explore the Create a Real-Time Dashboard documentation to build live dashboards from your monitoring data.
Fabric Capacities – Everything you need to know about what’s new and what’s coming
The Fabric Capacities team is excited to share details about the improvements we're making to the Fabric capacity management platform for Fabric and Power BI users. In this article we’ll cover: What are capacities? Great performance and simplified management with bursting and smoothing. Using Capacity Metrics to monitor usage and spend. Capacity platform and monitoring improvements. What are Capacities? Fabric is a unified data platform that offers shared experiences, architecture, governance, compliance, and billing. Capacities provide the computing power that drives all of these experiences. They offer a simple and unified way to scale resources to meet customer demand and can be easily increased with a SKU upgrade. Purchase a capacity once and use it to power everything Fabric Capacities are the foundation for simplicity and flexibility of Fabric's licensing model. With Fabric, you only need one capacity to drive all your Fabric experiences, without the hassle of provisioning many different services for every project. If you’re just getting started, Capacities can be acquired in three ways – the Fabric trial, Power BI Premium capacities, and Fabric capacities. If you have an existing Power BI Premium capacity you automatically have access to try out Fabric, you can learn about enabling fabric here. The 60-day Fabric free trial allows you to test out Fabric experiences with 64 CUs of throughput. You can then use the trial capacity to load-test and choose a SKU size to match demand. Selection of a capacity size determines the amount of capacity throughput, measured in Capacity Unit seconds (CUs). With the flexibility of shared compute, all for one fixed price, organizations can experience predictable spend on top of the flexibility of all the Fabric capabilities. On June 1, we announced the availability of Fabric pay as you go capacities for purchase in Azure. Fabric capacities come with a very low starting price point, complete pricing is available here. Additionally, reserved instance capacities will be coming in the near future that will provide even greater discounts when pre-committing. Great performance and simplified management with bursting and smoothing. Bursting for blazing performance running Fabric experiences. Bursting allows you to consume extra compute resources beyond what have been purchased to speed the execution of a workload. For example, instead of running a job on 64 CU and completing in 60 seconds, bursting could use 256 CUs to complete the job in 15 seconds. Bursting is a SaaS feature and requires no user management. Behind the scenes, the capacity platform is pre-provisioning Microsoft managed virtualized compute resources to optimize for maximum performance. Compute spikes generated from bursting will not cause throttling due to smoothing policies outlined in the next section. Smoothing helps streamline management by allowing you to plan for average usage instead of peak. When a capacity is running multiple jobs, a sudden spike in compute demand may be generated that exceeds the limits of a purchased capacity. Smoothing simplifies capacity management here by spreading the evaluation of compute to ensure that your jobs run smoothly and efficiently. For interactive jobs run by users: capacity demand is typically smoothed over 5 minutes to reduce short-term temporal spikes. For scheduled, or background jobs: capacity demand is spread over 24 hours, eliminating the concern of job scheduling or contention. Smoothing will not impact execution time, that is always at peak performance! Smoothing simply also allows you to size your capacity based on average, not peak usage. Using Capacity Metrics to monitor usage and spend As we announced in July, the extended preview period for Fabric will end on September 30th, 2023. Starting on October 1 st workload usage will count against capacity limits and capacity limits will be enforced. On October 1 st OneLake usage will also be charged on F SKU's. The Fabric Capacity Metrics app, announced in May’ 23 makes it easy to: Analyze Fabric experience resource consumption against capacity limits. View consumption metrics to plan for capacity scale-up or optimization. View OneLake consumption by capacity and workspace. Announcing Data Warehouse usage reporting for Capacity Metrics On September 18 th Capacity Metrics will support analysis of Fabric Data Warehouse usage reporting. This feature will let users measure the impact their organization’s Data Warehouse experiences against capacity limits for capacity planning and to better understand their compute spend generated from Data Warehouse experiences. Capacity Metrics will show Data Warehouse usage in the Warehouse Tab of the Items Table. Learn more in the Data Warehouse post on this update. Figure 3: Capacity Metrics with Data Warehouse usage Announcing OneLake storage reporting in Capacity Metrics We are excited to announce the availability of OneLake usage reporting in Capacity Metrics, starting on 9/18/2023. With this new feature, you can easily analyze your storage consumption by selecting your Capacity, choosing the date range, and viewing usage by workspace. This will provide you with valuable insights into your overall storage spend and enable you to monitor daily or hourly trends with usage of drill-through. Click here to view learn more about this feature. Figure 4: Capacity Metrics with OneLake storage analytics Capacity platform and monitoring improvements Starting on 10/1, we're improving the Capacity platform to handle larger and more diverse workloads. Here's what you can expect: Optimizations for long-running jobs: We're optimizing the platform for long-running jobs, so if a job exceeds capacity limits, it will run to completion (only subsequent jobs will be evaluated against limits) and the overage will be burned down against future capacity. Reduced throttling: We're introducing new policies to reduce throttling for customers who experience intermittent spikes in usage. Added overage protection: We're adding protection for large-scale jobs with automatic queue management to help prevent overloading of the capacity. Improved observability: The latest Capacity Metrics now has a throttling tab to help you monitor the new platform policies. Evolving the Capacity platform for longer running workloads We're introducing a new optimization for long-running jobs. Historically, if a job’s reported usage exceeded capacity limits, the following jobs would be throttled. Now, if a job’s reported usage exceeds capacity limits, throttling will not be immediately applied to following jobs. Instead, any overage will be automatically balanced against future capacity when the system has unutilized capacity. This feature to “borrow from the future” is in addition to smoothing and is also seamless to customers and supported by the following new analytics experience in Capacity Metrics. Updated throttling policies with overage protection Before the October 1st update, throttling occurs when smoothed usage is greater than 100% of the purchased capacity throughput. After the October 1 st platform update, capacity throttling policies will now be based on the amount of future capacity consumption that resulted from smoothing policies, this offers increased Overage protection for when future use is less than 10 minutes and richer queue management features to prevent excessive overload when usage exceeds an hour. The 4 new policies are outlined in Table 1. Future Smoothed Consumption – Policy Limits Platform Policy Experience Impact Usage <= 10 minutes Overage protection Jobs can consume 10 minutes of future capacity use without throttling. 10 minutes < Usage <= 60 minutes Interactive Delay User requested interactive type jobs will be throttled. 60 minutes < Usage <= 24 hours Interactive Rejection User requested interactive type jobs will be rejected. Usage > 24 hours Background Rejection User Scheduled background jobs will be rejected from execution. Table 1: Updated Capacity Throttling Policies To help you monitor and analyze the new policies, we've added a new throttling tab in the utilization section of the Capacity Metrics. You can now easily observe future usage as a percentage of each limit, and even drill down to specific workloads that contributed to an overage. Evolving Power BI Premium from v-cores to capacity units (CU) In the May ’23 post, we announced the rollout of capacity units as the unit of measurement for capacity throughput on Fabric. Capacity Units offer more granularity than the previously used v-cores and let us offer smaller sized capacities to Fabric customers with a very low entry point for pricing. Starting on 10/1, we will be updating all Power BI premium SKU’s (EM, P and A) to report in capacity units. Key takeaways for this change: This update will not result in any change to the throughput of a capacity. Power BI Premium SKU’s EM / A and P will now report usage using CUs. There will be one version of the Capacity Metrics app that supports all Power BI and Fabric capacity SKUs. See Figure 5: for an overview of capacity evaluation and throughput before and after the change. ities_Everything_you_need_to_know_about_whats_new_and_whats_coming Figure 5: Capacity Throughput, Measurement an Evaluation The change to consolidate the metrics units used for capacity analysis sets the stage for new cross-capacity analytics experiences for our customers who manage a large number of capacities. Next Steps On September 18th, please update to the latest version of the capacity metrics to get access to the latest OneLake analytics. Capacity Administrators can access Capacity Metrics directly from the Capacity settings page of Admin portal. We’ll also be releasing another update on October 1st that includes analysis for the new platform capabilities outlined above. The team’s super excited to share these platform and observability features to simplify management and administration of Fabric capacities. We look forward to your feedback and can’t wait to see all the amazing solutions you’ll create using Fabric experiences!Fabric May 2025 Feature Summary
Today kicks off Microsoft Build and we have a lot of new features in store for you. Some highlights are the Fabric Roadmap tool, a way to get glimpse of that is coming soon to Fabric. Chat with your data, powerful AI capabilities that make Power BI even easier. Cosmos DB in Fabric, give you the power of Cosmos DB that's AI-ready. To get a taste of the Build excitement be sure to check out Arun Ulag's Arun Ulag's announcement blog and Kim Manis' announcement blog. Contents Events & Announcements New Fabric Roadmap tool Fabric Platform Additional REST APIs for Fabric Deployment pipelines New capabilities for Fabric Git integration Shortcut transformations (Preview) Data Engineering New regions supported in User Data Functions SPN support for User data functions SPN support for the Livy API Private libraries support for User data functions Data Science Copilot in Power BI now supports Fabric data agents Fabric Data Agent Integration with Microsoft Copilot Studio (Preview) Data Warehouse Warehouse Snapshots (Preview) Real-Time Intelligence Call of the Cyber Duty: a new season of Kusto Detective Agency begins Continuous Ingestion from Azure Storage to Eventhouse (Preview) Fabric Eventhouse now supports Eventstream Derived Streams in Direct Ingestion mode (Preview) Get Data in Fabric Eventhouse from Lakehouse using OneLake Catalog Eventhouse Accelerated OneLake Table Shortcuts (Generally Available) Databases Introducing Cosmos DB in Microsoft Fabric (Preview) Native change data capture (CDC) support in Copy Job (Preview) Semantic Model Refresh Activity (Generally Available) Copilot for Data pipeline - boost your productivity in understanding and updating pipeline with Copilot Mirroring Mirroring for SQL Server On-Premises (Preview) Mirroring for SQL Server 2025 (Preview) New features for Mirroring for Azure SQL Managed Instance Customize retention period for mirrored data Mirroring region expansion Mirroring for Azure PostgreSQL region expansion Fabric Mirroring for Azure Cosmos DB: public preview refresh live with new features Dataflow Gen2 Dataflow Gen2 (CI/CD) (Generally Available) Dataflow Gen2 Public APIs (Preview) Dataflow Gen2 parameterization (Preview) Lakehouse as an incremental refresh destination in Dataflow Gen2 (Preview) SharePoint files as a destination in Dataflow Gen2 (Preview) Natural language to custom column Community Power Designer - unleash your inner report wizard (Generally Available) Closing https://youtu.be/5qbIn80JrqY?si=BxhW9e_0Ck7QN-1M Events & Announcements New Fabric Roadmap tool We’ve heard from you that it’s critical to know when key Fabric features will land, especially those that directly impact your use cases or unblock your organization’s adoption. For example, if you're waiting on Private Link support for Workspaces due to internal security requirements, you need a clear view of when that capability is planned and when it becomes available. Until now, this information was spread across Release Plan documentation pages. Today, we’re making that experience better. The new Fabric Roadmap page brings it all together in one place, with a cleaner interface, real-time updates, and direct integration with the internal planning tool used by the Fabric team. Check it out at https://roadmap.fabric.microsoft.com and tell us what you think in the comments. Power BI Some of the highlights include Chat with your data, a revolutionary new way to use AI in PowerBI. And Translytical task flows, enabling users to automate action directly within the report—streamlining decision-making and operational follow-through. To learn about all of the latest updates to Power Bi head over to the Power BI May 2025 Feature Summary Fabric Platform Additional REST APIs for Fabric Deployment pipelines An additional batch of Fabric public APIs for Deployment pipelines have been released, following our initial release of Deploy APIs a few months ago. With this new release, the full list of available Fabric APIs now matches the APIs available in Power BI, excluding Admin APIs, which will be added later. This marks a significant milestone in our ongoing efforts to enhance the Fabric platform and provide our users with powerful tools to manage their deployment processes more efficiently. Overview of the new APIs The new APIs offer a range of functionalities that streamline the deployment process, making it easier for teams to manage their content across different environments: Pipeline management: Create, update, and delete deployment pipelines with the new APIs. Stage management: Get and update the deployment pipeline stages. Deployments management: List deployment pipeline operations and get details of specific deployment pipeline operations. Workspace assignment management: Assign and unassign workspaces to and from stages. Roles assignment management: List deployment pipeline role assignments and get or delete specific role assignments. Support for Service Principal (SPN) All fabric Deployment pipelines REST APIs are now having the support for Service Principal (SP). This allows for more secure and automated deployments, enabling teams to integrate Fabric into their existing DevOps workflows seamlessly. Getting started To start using the new Fabric public APIs for Deployment pipelines, you can refer to the Automate your deployment pipeline with Fabric APIs documentation. This provides comprehensive guides and examples to help you integrate these APIs into your deployment processes effectively. New capabilities for Fabric Git integration Service Principal (SPN) support for Azure DevOps A few weeks ago, we announced the capability to use Service Principal when working with Fabric Git API and your Git Provider is GitHub. Soon, we will support Azure DevOps as your Git Provider as well. Cross-Tenant support for Azure DevOps Previously, connecting your workspaces using your identity to an Azure DevOps repository required both Fabric and your Azure DevOps organization to reside within the same tenant. However, we're thrilled to announce that this limitation will soon be a thing of the past. With our upcoming update, you'll be able to connect to an Azure DevOps organization even if it belongs to a different tenant than your Fabric tenant. Shortcut transformations (Preview) The preview of shortcut transformations introduces the ability to transform data as it’s shortcut into Fabric including changing the data format into Delta tables or applying AI transformations to unstructured data—such as summarizing text, translating content, or classifying documents. Data Engineering New regions supported in User Data Functions After our preview launch, we have been working on increasing the number of regions supported for this feature. We have recently added the following 14 new regions where you can use this feature from: Australia Southeast Brazil South Canada Central Central India France Central Korea Central North Central US Norway East South Africa North South India UAE North UK West West Europe West US You can find the entire list of supported regions in this article: Fabric Region Availability. This article will be frequently updated to reflect the latest region support. SPN support for User data functions Fabric User data functions now support Service Principal Names (SPN) to run functions. This feature allows organizations to ensure compatibility with enterprise identity and access management systems. By using SPNs, it is possible to implement applications that can call a user data function without requiring user credentials. This aligns with the zero-trust security model, providing a secure way where user data functions are the glue between your application and your data in Fabric. To learn more, refer to the SPN support for user data functions documentation. SPN support for the Livy API The Fabric Livy API for Data Engineering now supports Service Principal Names (SPN) to submit and execute Spark code. This added SPN authentication method allows organizations to ensure compatibility with enterprise identity and access management systems. By using SPNs, it is possible to implement applications that can call a user data function without requiring user credentials. This aligns with the zero-trust security model, providing a secure way where user data functions are the glue between your application and your data in Fabric. To learn more, refer to the Create and run Spark Session jobs using the Livy API documentation. Private libraries support for User data functions A new feature has been introduced: Private libraries support for Fabric user data functions. These private libraries are code created by you or your organization. Data engineering can be challenging, especially with data quality and complex analytics. Private libraries help streamline work and enable proprietary code use within a team securely. Fabric User data functions now allow custom library uploads in .whl format, containing scripts or modules for internal business logic. This can improve developer productivity across your organization allowing you to reuse these libraries for automating various process across different teams or departments in your organization. To learn more refer to the documentation on How to manage libraries for your Fabric User Data Functions. Data Science Copilot in Power BI now supports Fabric data agents Fabric data agents can be used in the new chat with your data experience in Power BI to get answers to your questions and explore your data more effectively. This integration enables users in Copilot in Power BI to not only connect Power BI semantic models, but also to a wider range of data sources in OneLake—such as lakehouses, warehouses, and KQL databases—retrieving insights seamlessly through Fabric data agents. When you ask a question in the new full-screen Copilot in Power BI experience Copilot first searches for relevant Fabric data agents you have access to. If you have the necessary permissions, it uses those data agents to retrieve answers based on your access rights. This helps you discover content, ask questions, perform quick analyses, and refine insights—all without switching tools or leaving Copilot. You can also manually add a data agent to your Copilot session and chat with it directly from Copilot in Power BI, enabling seamless access to your OneLake data. Fabric Data Agent Integration with Microsoft Copilot Studio (Preview) Fabric data agent will be available in preview and can be added as an agent to your custom setup in Microsoft Copilot Studio. With this integration, your custom agent can access data stored in Microsoft OneLake—including lakehouses, warehouses, Power BI semantic models, and KQL databases—and retrieve insights seamlessly through the Fabric data agent. Once you add the Fabric data agent to your custom agent, you can publish your custom agent to various consumption channels, including Microsoft Teams and Microsoft 365 Copilot, and share it with specific users or your entire organization. When a user asks a question from the custom agent in any of these channels, the Fabric data agent is used to retrieve answers—provided the user has the necessary permissions. Responses are always scoped to the user’s access rights, making it easier to discover relevant content, perform quick analyses, and refine insights within the same channel. To extend functionality, you can define actions for your custom agent. Actions such as sending emails or initiating other tasks allow the agent to automate processes on behalf of users, helping streamline workflows and improve productivity without leaving the custom agent experience. Data Warehouse Warehouse Snapshots (Preview) Ensuring data consistency during ETL (Extract, Transform, Load) processes has long been a challenge for data engineers. We are pleased to announce the preview of Warehouse Snapshots, a new feature in Microsoft Fabric designed to offer a stable, read-only view of your data warehouse at a specific point in time. This capability facilitates uninterrupted analytics and reporting. A warehouse snapshot is a read-only representation of a data warehouse at a designated moment, retained for up to 30 days (until configurable retention is available). Warehouse snapshots can be seamlessly ‘rolled forward’ on demand, enabling consumers to connect to the same snapshot (or use a consistent warehouse connection string from third-party tools) to access a curated version of data. This ensures that data engineers can provide analytical users with a consistent dataset, even as real-time updates occur. Analysts can run SELECT queries based on the snapshot without any ETL interference. For more information on CRUD for warehouse snapshots and understanding their considerations and limitations, please refer to Warehouse Snapshot in Microsoft Fabric (Preview). https://youtu.be/cUGGrdpswLk?si=03UTJCbjvTD8WwjU Real-Time Intelligence Call of the Cyber Duty: a new season of Kusto Detective Agency begins Are you ready to put your sleuthing skills to the test? The Kusto Detective Agency is back - and this time, it’s bigger, bolder, and packed with adrenaline. Introducing ‘Call of the Cyber Duty’, a brand-new season of the Kusto Detective Agency challenge designed for the sharpest minds in data. Whether you're a seasoned Kusto veteran or a curious newcomer, this is your chance to dive into a thrilling online race where speed, smarts, and strategy collide. Challenge begins June 8, 2025 Register by June 7, 2025 Why should you care? Because this isn’t just a challenge - it’s a competition. And the stakes, monumental. $10,000 for 1st place Bragging rights across the Fabric community Team up or go solo - form a squad of up to six detectives or take on the mission alone. Who should join? If you're using Microsoft Fabric Real-Time Intelligence and working with Eventhouse, this is your moment. The challenge is built to stretch your KQL muscles, sharpen your investigative instincts, and connect you with a vibrant community of data detectives. How to get started: Watch the trailer: Kusto Detective Agency - Call of the Cyber Duty Visit https://detective.kusto.io to register Rally your team or fly solo Prepare for a season of puzzles, plots, and powerful insights. This is more than a game. It’s a celebration of what’s possible with Kusto and Microsoft Fabric. So, gear up, detectives—the cyber world needs you. Disclaimer: No Purchase Necessary. Must be 14+ to participate. Registration period closes on June 7th, 2025, end of day. Prizes are awarded as digital gift cards to the team leader. Continuous Ingestion from Azure Storage to Eventhouse (Preview) Get Data in Real-Time Intelligence Eventhouse offers a step-by-step process to guide you through importing or inspecting the incoming data, creating or editing the destination table schema, to exploration of the ingested result from multiple sources. One of the sources from which users can bring data into an Eventhouse table using Get Data wizard is Azure Storage, which allows users to ingest one or more blobs/files from the storage account. This capability is now being enhanced with the feature of continuous ingestion, where once the connection between the Azure Storage Account and Eventhouse has been established, any new blob/file uploaded to the storage account will automatically be ingested to the destination table. Continuous Ingestion from Azure Storage to Eventhouse, utilizes Azure Events in Fabric to listen to Azure Storage Account Events. Based on the subscribed events from Azure Events, Eventhouse pulls the corresponding newly created/renamed file from the connected Azure Storage. This simplifies the process of bringing data from your Azure Storage account as it is being generated and eliminates the need for creating and maintaining long complicated ETL pipelines. It also eliminates the need of defining time-based triggers for fetching new data from Azure storage and makes ingestion to Eventhouse near real-time. Continuous ingestion from Azure Storage to Eventhouse is now offered in preview in Microsoft Fabric. To learn more, refer to the Get data from Azure storage documentation. https://youtu.be/RlZnGloBvSA?si=IU5YbxGG8Fombygu Fabric Eventhouse now supports Eventstream Derived Streams in Direct Ingestion mode (Preview) The Eventstreams feature in Microsoft Fabric Real-Time Intelligence allows you to bring real-time events into Fabric, transform them, and then route them to various destinations such as Eventhouse, without writing any code (no-code). You can ingest data from an Eventstream to Eventhouse seamlessly either from Eventstream artifact or using Eventhouse Get Data Wizard. This capability is now being extended to support Eventstream Derived streams in direct ingestion mode. Derived stream is a specialized type of destination that you can create after adding stream operations, such as Filter or Manage Fields, to an Eventstream. The derived stream represents the transformed default stream following stream processing. You can route the derived stream to multiple destinations in Fabric and view the derived stream in the Real-Time hub. Direct ingestion from Derived stream allows you to ingest your event data directly into the Eventhouse without any processing. This can be configured from Eventstream, as well as from Eventhouse Get Data Wizard, including embedded Real-Time Hub in Eventhouse Get Data Wizard. Please refer to Get data from Eventstream to learn more and get started today. Get Data in Fabric Eventhouse from Lakehouse using OneLake Catalog OneLake catalog is the central hub for discovering and managing Fabric content. One of the artifacts that OneLake catalog enables the discovery of is Microsoft Fabric Lakehouse, which is a data architecture platform for storing, managing, and analyzing structured and unstructured data in a single location. Get Data in Eventhouse now embeds OneLake catalog which allows an easy discovery and navigation experience for ingesting data from Lakehouse to Eventhouse. Using OneLake catalog, you can easily look for a Lakehouse through multiple workspaces and identify the Lakehouse you recently used, your favorites or endorsed by your organization. Once you select Lakehouse from the embedded OneLake catalog in Eventhouse Get Data, you can select and ingest a file from the Lakehouse seamlessly, including the files within sub folders. To learn more, refer to the Get data from OneLake documentation and get started! Eventhouse Accelerated OneLake Table Shortcuts (Generally Available) Shortcuts are embedded references within OneLake that point to other files’ store locations without moving the original data. Previously, you could create a shortcut to OneLake delta tables using Eventhouse and query the data, but performance lagged direct ingestion in Eventhouse, as shortcut queries lacked the powerful indexing and caching capabilities of Eventhouse. Accelerated shortcuts are powered by query acceleration which indexes and caches data landing in OneLake on the fly, allowing customers to run performant queries on large volumes of data. Customers can use this capability to analyze real-time streams coming directly into Eventhouse and combine it with data landing in OneLake either coming from mirrored databases, Warehouses, Lakehouses or Spark. Customers can expect significant improvements by enabling this capability, in some cases up to 50x and beyond. How to enable Query Acceleration? You will now see an option to enable Acceleration while creating a new shortcut from Eventhouse. To learn more, refer to the Query acceleration for OneLake shortcuts - overview (preview) documentation. Databases Introducing Cosmos DB in Microsoft Fabric (Preview) Cosmos DB is now available in preview as a new addition to the databases workload in Microsoft Fabric. Cosmos DB in Fabric is easy to set up, with automatic scale and secure by default, enabling you to build AI applications with less overhead. You can store and retrieve semi-structured data within milliseconds, without having to tweak the database settings manually. Equipped with built-in vector indexing and AI-ready full-text, hybrid search capabilities of Cosmos DB, you can now seamlessly build GenAI applications. Your existing or new applications can instantly benefit from deep integration with Fabric OneLake, bringing you databases, analytics, data science, real-time intelligence, and Copilot-powered BI in one place, rather than assembling them individually. You can seamlessly join Cosmos DB data with any other data in OneLake, such as SQL DB, truly unifying your data estate. To get started, please join our preview program by filling in this opt-in form. For more information, refer to Announcing Cosmos DB (Preview). Data pipelines Native change data capture (CDC) support in Copy Job (Preview) Change Data Capture (CDC) in Copy Job is a powerful capability in Data Factory that enables efficient and automated replication of changed data including inserted, updated and deleted records from a source to a destination. This ensures your destination data stays up to date without manual effort, improving efficiency in data integration while reducing the load on your source system. With CDC in Copy Job, you can enjoy the following benefits: Zero Manual Intervention: Automatically captures incremental changes (inserts, updates, deletes) directly from the source. Automatic Replication: Keeps destination data continuously synchronized with source changes. Flexible Incremental Copy Options: Automatically detects CDC-enabled tables, allowing you to choose between CDC-based or watermark-based incremental copy at the table level. Optimized Performance: Processes only changed data, reducing processing time and minimizing load on the source. Learn more in the What is Copy job in Data Factory documentation. Semantic Model Refresh Activity (Generally Available) Semantic Model Refresh activity for data pipelines is now generally available! With this activity, you will be able to create connections to your Power BI semantic model datasets and refresh them from your data pipeline! To learn more, refer to the Semantic model refresh activity in Data Factory for Microsoft Fabric documentation. Copilot for Data pipeline - boost your productivity in understanding and updating pipeline with Copilot Maintaining complex Data pipelines in your ETL project is not easy work, especially when you need to understand complicated Data pipelines created by others or you need to update some configurations for a set of pipelines. Copilot for Data pipeline helps users quickly understand the purpose of a pipeline and the details of its activities. With this new release, it also allows users to update descriptions for pipelines and activities based on its summary. After updating, users can hover over any activity to see a simple explanation of its function. Copilot for Data pipeline empowers users to efficiently update the settings of multiple activities within seconds which is much faster than manual update that may take hours. For example, users can update the timeout of more than ten activities inside pipeline from 12 hours to 1 hour. To learn more, refer to the AI-powered development with Data pipeline documentation. Mirroring Mirroring for SQL Server On-Premises (Preview) Mirroring for SQL Server in Fabric for on-premises versions of SQL Server 2016-2022 is now in Preview! Mirroring in Fabric allows users to enjoy a highly integrated, end-to-end, and easy-to-use product that is designed to simplify your analytics needs. Built for openness and collaboration between Microsoft, and technology solutions that can read the open-source Delta Lake table format, Mirroring is a low-cost and low-latency turnkey solution that allows you to create a replica of your SQL Server data in OneLake which can be used for all your analytical needs. By leveraging Change Data Capture (CDC) technology available in SQL Server, mirroring service in Fabric uses on-premises data gateway (OPDG) to connect to SQL Server and read the initial snapshot as well as subsequent changes to data at the source. OPDG then pulls the data into OneLake and converts into an analytics-ready format in Fabric. To learn more, refer to the Mirroring for SQL Server in Microsoft Fabric (Preview) blog post. Mirroring for SQL Server 2025 (Preview) With the announcement of Microsoft SQL Server 2025 at Build, customers can also leverage mirroring from this version in Fabric. The overall user experience is like mirroring from other SQL Server versions and Azure SQL. Mirroring for SQL Server 2025 uses change feed instead of Change Data Capture and SQL Server keeps track and replicates the initial snapshot and changes to the landing zone in OneLake which is then converted to an analytics-ready format by the mirroring engine. To learn more, refer to the Mirroring for SQL Server in Microsoft Fabric (Preview) blog post. 025_Feature_Summary New features for Mirroring for Azure SQL Managed Instance We have made substantial updates to Mirroring for Azure SQL Managed Instance in Fabric. Based on user feedback, new features have been developed to address data replication needs: Mirror Azure SQL Managed Instance via private endpoint: VNet data gateway or on-premises data gateway can be used as a way to connect to your Azure SQL Managed Instance database for mirroring, removing the necessity of opening public access. The data gateway ensures secure connections to the source databases via private endpoint. Mirror tables without Primary Keys: We’ve relaxed the limitation to let you mirror tables even if they don’t have a primary key sometimes referred to as heap tables, offering increased flexibility. Support for expanded Data Definition Language (DDL): In addition to Alter/Drop/Rename tables/column, now you can Truncate tables in the source databases while mirroring is active. To learn more, refer to the Mirrored Databases from Azure SQL Managed Instance documentation. Customize retention period for mirrored data Mirroring in Fabric continuously replicates your existing data estate from various databases into OneLake in Delta Lake table format. To keep the mirrored data efficiently stored and always ready for analytics, mirroring automatically runs vacuum to remove old files no longer referenced by a Delta log. We now offer you the flexibility to customize the retention setting according to your requirements. For instance, you may choose a shorter retention period to reduce mirroring storage consumption or extend the retention period to utilize Delta’s time travel capabilities for analytics. Currently, this value can be set via API. To learn more, refer to the retention for mirrored data documentation. Mirroring region expansion Mirroring now supports all regions that are available for workloads in Microsoft Fabric. We have recently added a new region support for West US 3 to meet the growing customer demand. For detailed information about the Fabric regions that support mirroring, please refer to the supported regions documentation. Mirroring for Azure PostgreSQL region expansion Alongside the region expansion for all Mirroring in Fabric, Mirroring for Azure PostgreSQL will also expand region support from the initial 4 regions: Canada Central, West Central US, East Asia, and North Europe to all regions supported by Mirroring in Microsoft Fabric to ensure that customers have the best performance when replicating data from Azure PostgreSQL flexible server. To learn more, refer to the Simplify Your Data Strategy: Mirroring for Azure Database for PostgreSQL in Microsoft Fabric for Effortless Analytics on Transactional Data. Fabric Mirroring for Azure Cosmos DB: public preview refresh live with new features We’re thrilled to announce the latest refresh of Fabric Mirroring for Azure Cosmos DB! This update introduces key enhancements like Microsoft Entra ID authentication, container selection, support for special characters in column names, and even vector search compatibility for AI workloads. With features like auto schema inference and full CRUD API support, this release makes it easier than ever to build secure, scalable, and real-time analytics pipelines with Cosmos DB data in OneLake. To learn more, refer to the Fabric Mirroring for Azure Cosmos DB with new features blog post. Dataflow Gen2 Dataflow Gen2 (CI/CD) (Generally Available) With this new set of features now generally available, you can seamlessly integrate your Dataflow with your existing CI/CD pipelines and version control of your workspace in Fabric. This integration allows for better collaboration, versioning, and automation of your deployment process across dev, test, and production environments. New Dataflow Gen2 item experience with the option to enable Git integration, deployment pipelines and Public API scenarios. Key benefits Automated deployments: streamline your deployment process by integrating Dataflow with your CI/CD processes in Fabric. Version control: use GIT to manage and version your Dataflow Gen2, ensuring you have a history of changes and can easily roll back if needed. Collaboration: enhance team collaboration by leveraging GIT’s branching and merging capabilities. Multitasking support: you can now have multiple Dataflows open at the same time as other Microsoft Fabric experiences. These new features will significantly improve your workflow and productivity when working with Dataflows Gen2 in Fabric. We look forward to hearing your feedback and suggestions as we continue to enhance this feature. To learn more, refer to the Dataflow Gen2 with CI/CD and Git integration support (Preview) documentation. Dataflow Gen2 Public APIs (Preview) Data Factory in Fabric now provides a robust set of APIs that enable users to automate and manage their dataflows efficiently. These APIs allow for seamless integration with various data sources and services, enabling users to create, update, and monitor their data workflows programmatically. The APIs support a wide range of operations -- including dataflows CRUD (Create, Read, Update, and Delete), scheduling, and monitoring -- making it easier for users to manage their data integration processes. The APIs for dataflows in Fabric Data Factory can be used in various scenarios: Automated deployment: Automate the deployment of dataflows across different environments (development, testing, production) using CI/CD practices. Monitoring and alerts: Set up automated monitoring and alerting systems to track the status of dataflows and receive notifications in case of failures or performance issues. Data integration: Integrate data from multiple sources, such as databases, data lakes, and cloud services, into a unified dataflow for processing and analysis. Error handling: Implement custom error handling and retry mechanisms to ensure dataflows run smoothly and recover from failures. Learn more about Dataflow APIs in the documentation. Dataflow Gen2 parameterization (Preview) Parameters in Dataflow Gen2 enhance flexibility by allowing dynamic adjustments without altering the dataflow itself. They simplify organization, reduce redundancy, and centralize control, making workflows more efficient and adaptable to varying inputs and scenarios. Leveraging query parameters while authoring Dataflows Gen2 has been possible for a long time, however, it was not possible to override the parameter values when refreshing the dataflow. The ability to pass values from a pipeline into a Dataflow parameter for refresh has been one of the top ideas in the Fabric ideas portal since Dataflow Gen2 was released. We are happy to announce the preview of the public parameters capability for Dataflow Gen2 with CI/CD support as well as the support for this new mode within the Dataflow refresh activity in Data pipelines. Public parameters in Dataflow Gen2 with CI/CD support allow users to refresh their Dataflows by passing parameter values outside of the Power Query editor through the Fabric REST API or native Fabric experiences. This enables a dynamic experience with Dataflows, where each refresh can be run with different parameters that affect how the Dataflow is refreshed. To learn more about this new feature, refer to the Use public parameters in Dataflow Gen2 (Preview) documentation. Lakehouse as an incremental refresh destination in Dataflow Gen2 (Preview) Incremental refresh for Lakehouse destinations in Dataflow Gen2 is now in preview! This feature introduces a powerful way to optimize performance and ETL pipeline for one of the most popular data destinations to date. With incremental refresh, users can ensure faster refresh cycles, improve system efficiency, and reduce resource consumption, making it an ideal solution for large-scale analytics and operational data scenarios. This functionality is particularly valuable for businesses leveraging Lakehouse centric solutions to consolidate structured and unstructured data into a unified data model. To use this capability, configure your Dataflow Gen2 with a Lakehouse destination and enable incremental refresh settings within your dataflow editor as usual. Make sure to check out our documentation here to learn more about the considerations when you are using Lakehouse as a destination. To learn more, refer to the Incremental refresh in Dataflow Gen2 documentation. SharePoint files as a destination in Dataflow Gen2 (Preview) SharePoint data destinations in Dataflows Gen2 is now in preview! This innovative feature empowers users to seamlessly write CSV files directly into their designated SharePoint sites, streamlining data integration and enhancing team collaboration within Office 365. Using this new capability, users can effortlessly configure their dataflow queries to output data into specific folders within SharePoint, facilitating smoother workflows and ensuring that your data remains accessible and actionable in your operational processes. We encourage you to explore the possibilities of this feature and provide valuable feedback to help us refine and expand its functionality. Stay tuned for more updates and improvements as we continue to evolve data destinations for Dataflows Gen2! To start using SharePoint data destinations in Dataflows Gen2, follow these simple steps: Create a Dataflow Gen2 Get some data from one of your data sources In the destination settings, choose SharePoint as your output location. Provide the URL of the specific SharePoint site where you want your CSV files to be saved. Make sure you select the correct authentication method. Execute the dataflow to generate and store your CSV files in the selected SharePoint destination. To learn more, refer to the Dataflow Gen2 data destinations and managed settings documentation. Natural language to custom column Copilot is now available within the Custom column dialog of Dataflow Gen2. You can leverage a new Copilot experience where you can have Copilot write a custom column formula based on a prompt that you provide. For example, for a table that has the fields **OrderID**, **Quantity**, **Category**, **Total** you can pass a prompt like the following: If the total order is more than 2000 and the category is B, then provide a discount of 10%. If the total is more than 200 and the category is A, then provide a discount of 25% but only if the quantity is more than 10 otherwise just provide a 10% discount. After submitting this prompt, Copilot will process it and modify the custom column formula for you and adding a name and a data type if necessary. Be sure to give this new Copilot experience inside Dataflow Gen2 a try and share your feedback with us. Community Power Designer - unleash your inner report wizard (Generally Available) PowerBI.tips, in collaboration with Microsoft Fabric, is thrilled to announce that Power Designer is now Generally Available! This is a real time-saving application you won’t want to miss! Transition from generic, standard reports to sophisticated and highly customized presentations. What is Power Designer about? Power Designer is sleek, intuitive, and fun, making designing reports feel less like work and more like unleashing your inner artist. Craft themes like a pro: create detailed theme files for your Power BI reports with ease. Customize colors, fonts, and styles to match your brand. Real-Time visuals: watch your Power BI visuals update live as you build your style. Multipage mastery: add background images to each page with a snap, transforming your reports into polished, magazine-worthy layouts. AI-Powered: let AI take the wheel with auto-placement of visuals in your multipage templates. Preview: test your new theme on reports already published in your workspaces with the preview feature. Now that Power Designer has been released, it’s time to jump in and start creating. Head to your Fabric Workspaces, fire up Power Designer, and let your imagination run wild. To learn more, refer to PowerBI.tips Designer Now in Fabric – Power Designer. Check out the YouTube Video: Introducing Power Designer: Unleash Your Inner Report Wizard! Closing We hope that you enjoy the update! Be sure to join the conversation in the Fabric Community and check out the Fabric documentation to get deeper into the technical details. As always, keep voting on Ideas to help us determine what to build next. We are looking forward to hearing from you!
Materialized Lake Views in Microsoft Fabric (Generally Available)
If you haven’t already, check out Arun Ulag’s hero blog “FabCon and SQLCon 2026: Unifying databases and Fabric on a single, complete platform” for a complete look at all of our FabCon and SQLCon announcements across both Fabric and our database offerings. Since introducing MLVs (Preview) at Build 2025, data engineers have used them to replace hand-built ETL pipelines with a few declarative Spark SQL statements, and their feedback directly shaped this release. This update closes the most important gaps since reaching preview and makes MLVs production-ready at scale. With multi-schedule support, broader incremental refresh, PySpark authoring, in-place updates, and stronger data quality controls, teams can now build, run, and evolve medallion pipelines with far less operational overhead. What are Materialized Lake Views? A materialized lake view in Fabric is a persisted, automatically refreshed view defined in Spark SQL or PySpark. It enables express multi-stage Lakehouse transformations, typically referred to as medallion architecture in the bronze-to-silver-to-gold pattern as declarative statements rather than custom Spark jobs. Fabric tracks dependencies between MLVs, orchestrates refreshes in the correct order, and enforces data quality constraints at every stage. The result is a complete medallion pipeline you can set up in minutes and monitor from a single pane. Figure: Materialized lake views make it easier to implement medallion architecture on Fabric and make your pipelines production ready. Broader clause coverage for optimal refresh Processing only what’s changed instead of recomputing an entire view has been a core promise of MLVs since preview. Optimal refresh covers far more of the queries data engineers write every day. MLVs can now refresh incrementally when the definition includes: Aggregations such as COUNT and SUM with GROUP BY. Left outer joins, left semi joins. Common table expressions. These additions mean that most real-world medallion pipelines qualify for incremental processing without any rewriting. You don’t need to decide when incremental refresh applies. With optimal refresh, a built-in decision engine examines each refresh, evaluates the volume of changed data against the cost of a full recomputation, and automatically chooses the faster path. Change Data Feed is enabled by default on every new MLV, so there is nothing to configure. The result is straightforward: as your data grows, refresh times stay predictable and compute costs stay low. PySpark authoring support for MLVs (Preview) With PySpark support, data engineers can now create, refresh, and replace MLVs directly from Fabric notebooks using PySpark and the familiar DataFrameWriter API. Data engineers reach for PySpark when their work goes beyond what Spark SQL can express cleanly — applying custom cleansing logic with Python libraries, calling user-defined functions that wrap business rules or ML models, or scheduling multi-step transformations that mix procedural code with DataFrame operations. These patterns are common in production Lakehouse pipelines, and PySpark authoring now brings them into the MLV framework. Figure: New PySpark (Python) notebook option for creating materialized lake views. PySpark MLVs support: Data quality constraints (including expression-based rules and session-scoped UDFs). Table properties. Scheduled refreshes from the same notebook where data is prepared and explored. The entire pipeline, from raw ingestion through a production-quality gold layer, can now live in one place. Today, PySpark MLVs perform a full refresh on each run. Optimal refresh support for PySpark-authored MLVs is coming soon. For a step-by-step walkthrough, see the PySpark MLV documentation. Multi-Schedule Support Previously, users were able to refresh all MLVs in a lakehouse on a single schedule. Teams with multiple data products at different cadences often worked around this with notebooks. This approach is error-prone and bypasses dependency management, centralized error reporting, and retry logic. For example, notebook-triggered refreshes do not surface MLV error details; failures appear only in the cell output, and dependent views have no awareness of them. Errors can persist week after week without anyone knowing the pipeline is broken. Now, multi-schedule support removes that complexity. You can now define named schedules within a lakehouse, each targeting a specific subset of views. For example: A finance pipeline can refresh the gold layer hourly. A lower-priority analytics pipeline can run every six hours. No custom scripting is required. When a named schedule runs, Fabric refreshes all upstream dependencies in the correct order, executes independent views in parallel, and surfaces errors centrally so issues don’t go undetected. If a run is already in progress when a schedule fires, the new run is skipped, and the next window proceeds as expected. Figure: Multiple independent schedules can now be configured for materialized lake view runs within a single lakehouse. Tip: Notebook-triggered refreshes don’t provide full dependency awareness or centralized visibility. Use Managed MLVs in Lakehouse for automatic dependency management, retries, and automatic monitoring. In-place view updates with Replace Business logic changes. A filter condition shifts, a join gains a new column, an aggregation adds a metric. Previously, updating an MLV meant dropping and recreating it from scratch, losing refresh history and forcing downstream consumers to reconnect. With Replace, you can update an MLV's definition in place. Fabric validates the new logic, swaps it in, and preserves the view's identity, metadata, and lineage. Downstream dependencies remain intact. Replace works for both SQL and PySpark-authored MLVs. Stronger data quality rules This update significantly expands data quality enforcement. In preview, constraints could check whether a column was null or matched a fixed value. For PySpark-authored MLVs, constraints can now: Use expression-based logic combining multiple columns. Apply arithmetic and built-in functions in a single rule. Invoke session-scoped user-defined functions for validation logic that lives in Python rather than SQL. Fabric tracks every constraint across every refresh and surfaces the results in a data quality report. You can quickly spot which rules fail most often, which views they affect, and how trends shift over time without building a separate monitoring pipeline. What’s ahead This is a milestone, not the finish line. We are actively working on optimal refresh for PySpark-authored MLVs, expanded optimal refresh coverage for more SQL operators, and deeper integration with other Fabric workloads. The roadmap is shaped by your feedback. Share your ideas on the Fabric Ideas portal and help influence what comes next. Get started Materialized Lake Views are available today in every Microsoft Fabric workspace. To start building: Refer to the materialized lake view documentation for quick starts and API reference. Try the end-to-end medallion architecture tutorial combining MLVs with Shortcut Transformations. Submit feedback and feature requests on the Fabric Ideas portal. Join us at FabCon & SQLCon 2026 in Atlanta, March 16–20, for hands-on sessions and deep dives.The future of conversational analytics in Fabric
What if you could ask questions about your business data in plain language, and get trusted answers wherever you work? Discover how Fabric IQ, Power BI, Microsoft 365 Copilot, and Fabric data agents are bringing the full breadth of conversational analytics into the flow of work, turning governed business context into insights and action.Fabric March 2025 Feature Summary
Welcome to the March Feature Summary! From the innovative Variable library (Preview) to the powerful Service Principal support in the CI/CD features, there's a lot to explore. Dive in and discover how the new Partner Workloads in Fabric bring cutting-edge capabilities to your workspace. Plus, enhanced OneLake security ensures your data is protected. And don't miss out on the expanded regional availability for Eventstream's managed private endpoints, making it easier for organizations worldwide to build secure, scalable streaming solutions. With FabCon kicking off today, the announcements are rolling in! Get ready to explore these features and more in the March 2025 updates for Fabric! Contents Power BI Fabric Platform Variable library (Preview) New CI/CD features New Partner Workloads in Fabric Workload Development Kit improvements Drive data discovery & curation with tags (Generally Available) Fabric Domains OneLake catalog and Modern Get Data are now integrated into Excel for Windows Enhanced search on tables and columns in the OneLake catalog New quick action for seamless navigation in the OneLake catalog Purview Data Loss Prevention (DLP) policies for KQL and Mirrored DBs Multi-tenant organization (MTO) (Generally Available) OneLake OneLake security OneLake shared access signatures (SAS) (Generally Available) Data Engineering Write capabilities and PySpark support in Spark Connector for Fabric DW Esri’s ArcGIS GeoAnalytics integration with Microsoft Fabric Spark (Preview) Deployment pipeline inside Spark Job Definition (Preview) Row-level and Column-level security in Spark Introducing Pylance language support for Fabric Notebook Environment sharing across workspaces (Preview) Shortcuts now supported in Lakehouse Git metadata representation and in Fabric Deployment pipelines Announcing Fabric User Data Functions (Preview) OPTIMIZE FAST and FSCK commands in Fabric Runtime 1.3 for Apache Spark (Generally Available) Fabric Spark Monitoring APIs (Preview) Notebook Integration with User Data Functions (UDFs) (Preview) Data Science Copilot in Notebooks Agentic and UX enhancements Fabric data agent Copilot and AI capabilities now available across all paid SKUs Fabric data agent integration with Azure AI Agent Service (Preview) Fabric data agent SDK (Preview) Data Warehouse AI functions in Data Warehouse Fabric User Data Functions in Data Warehouse Scalar SQL User-defined Functions Developer Experiences in Fabric Warehouse JSON data in OPENROWSET SQL audit logs (Preview) Fabric Data Warehouse item permissions OneLake Security for Lakehouse Analytics SQL Endpoints Private Preview Real-Time Intelligence Eventstream sources: MQTT, Solace, ADX, weather & Azure Event Grid Eventstream CI/CD & REST APIs (Generally Available) Expanded regional availability for Eventstream's managed private endpoints (Secure Outbound) Connect to Eventstream using Microsoft Entra ID authentication Preview Real-Time Data Streams for Apache Kafka, Confluent Cloud, Amazon MSK & Amazon Kinesis Continuous improvements to Eventhouse Get Data Experience Build event-driven workflows with Azure and Fabric Events (Generally Available) Eventhouse OneLake availability now supports backfill Improved Activator alerts from Power BI End-to-end Real-Time Intelligence samples Synapse Data Explorer to Eventhouse migration tooling (Preview) Data Factory Enterprise readiness VNET Gateway support for Data pipelines New and updated Certified Connectors for Power BI and Dataflows Simplifying Data Ingestion with Copy Job Mirroring Mirroring for Azure SQL Database protected by a firewall(Preview) Mirroring for Azure Database for PostgreSQL Flexible Server (Preview) Open Mirroring UX improvements Transformations Save a new Dataflow Gen2 with CI/CD support from a Dataflow Gen1, Gen2, or Gen2 (CI/CD) Incremental Refresh for Dataflow Gen2 and new support to Lakehouse as destination with incremental refresh (Generally available) Check ongoing validation status of a Dataflow Gen2 with CI/CD support Orchestration Apache Airflow Job (Generally Available) OneLake file triggers for pipelines Variable libraries for pipelines (Preview) Spark Job Definition pipeline activity parameter support Azure Databricks jobs activity now supports parameters User data functions in Data pipelines (Preview) Data Factory pipelines now support up to 120 activities Dataflow Gen2 Dataflow Gen2 with CI/CD capabilities Data pipelines Mounting ADF (Preview) Mirrored database (Generally Available) Copy Job (Generally Available) Parameterization Parameterized connections in Data pipelines Table Name parameter support for data destinations AI-powered experiences Efficiently build and maintain your Data pipelines with enhanced capabilities for Copilot in Data Factory Data integration shared experiences Partner workloads Workloads (Generally Available) Power BI Designer (Generally Available) Newly released workloads Profisee MDM Workload Lumel PowerTables Workload SAS Decision Builder Workload (Preview) Striim SQL2 Fabric Workload Closing Power BI This month, we're excited to introduce a range of new features and improvements that will elevate your data analysis and visualization experience. Among the highlights are the Copy report object name feature, which simplifies locating and identifying objects within the PBIR folder, and the better storytelling with Data annotations in Power BI for PowerPoint, allowing you to add descriptive text directly to visualizations in your presentations. Additionally, we've made significant enhancements to Reference Lines, enabling you to add shade areas for all reference line types and support reference lines on the Y-axis for Line and stacked column charts. The Category enhancements for new cards bring new styles for categories, including table style and cards style, with conditional formatting options. Dive in to explore these exciting features and see how they can help you make the most of your data. To find out more about these features and more, head over to the Power BI March 2025 Feature Summary. https://youtu.be/2ft7aZKnaXY?si=_5tz-qmlC6NiL4nF Fabric Platform Variable library (Preview) We are excited to announce the upcoming preview of a new CI/CD feature - Variable library item in Microsoft Fabric. This feature is designed to provide a unified and centralized way to manage configurations, reducing the need for hardcoded values and simplifying your CI/CD processes, making it easier to manage configurations across different environments. What is the Variable Library? The Variable library is a new item type in Microsoft Fabric that allows users to define and manage variables at the workspace level, so they could soon be used across various workspace items, such as data pipelines (already available!), notebooks, Shortcut for lakehouse and more. Key features and benefits Environment-specific configurations: With Variable library, you can define different sets of values for your variables, e.g. one for each stage of your release pipeline. This means you can easily switch configurations based on the deployment environment, such as development, testing, and production. Centralized management: The Variable library provides a centralized location to manage all your configuration variables. This makes it easier to update and maintain configurations, ensuring consistency across your deployments. 2. Integration with CI/CD pipelines: The Variable library integrates seamlessly with your CI/CD practices – It’s a fabric item which is supported in Git integration and Deployment pipelines, and it has APIs to automate its management. 3. Support for multiple Variable types: The Variable library supports various variable types, including boolean, integer, number, string, GUID, and DateTime. This flexibility allows you to define and use variables that best suit your needs. Fabric items supporting Variable library The Variable library is supported soon through various fabric items: Data pipeline – Where Variable library can be used in dynamic content fields. Notebook – Which will support using variables in Notebook code natively. Shortcut for Lakehouse – Where Variable library will be used to parameterize the shortcut configuration. This includes managing connections to data sources and defining paths. More supporting items are underway, so stay tuned! The new CI/CD feature of the Variable library will be available in early April in Microsoft Fabric and requires admin approval. Try it out starting mid-April 2025 and be part of this exciting journey! What is Fabric Variable library? To learn more, refer to the Variable library documentation. New CI/CD features In addition to Variable library, the CI/CD platform is releasing a few important updates to improve the developer experience when setting up your CI/CD process in Fabric. Service Principal support The following set of APIs will start supporting Service Principal as well: Deployment pipelines APIs GitHub (through Git APIs) Calling Git APIs when working with Azure DevOps as your git provider is still being worked on and will be released in the upcoming few months. Please stay tuned and thank you for your patience! If you want to learn more about how to automate your CI/CD process in Fabric, you can use one of the following resources: Automate Git integration by using APIs Automate deployment pipeline by using Fabric APIs fabric-cicd Branch out to existing workspace When working in Fabric using Source control, we recommend working on your own feature branch in an isolated environment. In Fabric, this means you need another workspace. We have made this process easy with the ability to ‘branch out’, landing you directly in a new workspace, already connected and synced to the new branch. Now, we are making things even easier, as you can branch out to an existing workspace. If you have your own developer workspace, you don’t need to create another one to work on your next task. You can simply choose the same workspace, which already has all settings configured and data in place and continues working instantly after connecting it to the new branch. New Partner Workloads in Fabric Our amazing Fabric partners are delivering new capabilities fully integrated with Fabric as Workloads! This allows for the creation of new item types in shared workspaces for team collaboration. The Workload Hub is Fabric’s in-product marketplace for Partners that natively integrated with Fabric to provide our community with the ability to try and purchase leading data applications performing from data storage, transformation and connectivity tools to MDM platforms and visualization - all in the native Fabric experience we know and love! Add a workload in the workload hub describes how customers can add and manage workloads that have been published as Fabric Workloads. Check out the Partner Workloads section in this blog to learn which workloads were released in the past month. Workload Development Kit improvements The Microsoft Fabric Workload Development Kit is designed to enhance the Microsoft Fabric experience by integrating custom capabilities into Fabric. It allows developers and Microsoft Partners to create and publish workloads providing a seamless user experience without leaving Fabric. By using the Workload Development Kit, developers can embed new capabilities in Microsoft Fabric, streamline analytics processes, and explore new avenues for revenue generation. We are introducing several new functionalities that will empower our community to build more integrated workloads for Fabric independently. OneLake integration Workloads can now leverage the new OneLake integration, which allows for storing both structured and unstructured data directly as part of the partner workload item. This integration enables customers to access the data through standard OneLake APIs and expose it as a data item in the OneLake catalog. Importantly, all customer data is stored and protected within the customer tenant. Enhanced navigation experience We have improved the navigation experience over the Workload Development Kit. The community can now build workloads that open new tabs and navigate directly to other items within the workspace, providing a smoother and more intuitive user experience. Promoting Workload Solutions In response to requests from workload developers, we are excited to introduce support for embedding videos on the workload page. Additionally, we have rolled out new Fabric monetization guidelines that the community can utilize as part of the Fabric UX system. Real-Time Intelligence integration For workload developers, we have extended our example to include Real-Time Intelligence. Partners can now use the Event House selector in their workloads to offer customers rich real-time experiences. We have also included an example of how to use the real-time APIs and execute queries against the Event House. There are several additional changes that are helping the community build new workloads. Be sure to check out the Workload Development Kit - Announcing OneLake support and Developer Experience enhancements to learn more. Drive data discovery & curation with tags (Generally Available) Tags in Fabric enable flexibility in how you structure and manage your data estate and are now generally available. By providing the ability to apply additional metadata to items in Fabric, tags help admins and data owners categorize the data, enhancing the searchability and boosts success rates and efficiency for end users. To learn more on Tags in Microsoft Fabric refer to our documentation. In the OneLake catalog, the tagging experience has been further refined with context-aware applied tags. Now, when users filter data by tags, they will only see relevant tags applicable to their current context instead of browsing through the entire organization’s tag collection. This enhancement reduces clutter and improves efficiency when searching for tagged assets. Fabric Domains Microsoft Fabric’s data mesh architecture supports organizing data into domains & sub domains helping admins to manage and govern the data per business context with various delegated settings. Domains & sub domains structure enables data consumers to filter and discover content from the area most relevant to them. We have improved the visibility of the selected domain within OneLake catalog and enriched the domain image gallery with new, vivid imagery. Now, when users filter by domain in OneLake catalog they'll see the domain's cover image displayed in the background. This will create more clarity for users in their current context as they browse the catalog. Coming soon - create Tags in domains Domain admins will soon be able to create a list of tags in their domain. Item owners will be able to apply these tags to their items within the domain and data consumers will be able to use them to filter and search relevant data. OneLake catalog and Modern Get Data are now integrated into Excel for Windows It’s now even easier than ever to work with your Fabric data in Excel! The OneLake catalog is now integrated into the Modern Get Data experience in Excel, allowing users to effortlessly discover and connect to their Lakehouse or Warehouse assets. With just a few clicks, you can bring Fabric data into Excel for analysis and decision-making. This integration is currently available to customers enrolled in the M365 Insiders program on the Beta Channel (Insiders Fast), with plans for a broader rollout soon. For more information, refer to Announcing a new modern data connectivity and discovery experience in Dataflows. Enhanced search on tables and columns in the OneLake catalog Navigating through your data just got more efficient! Our new enhanced search capability allows users to search for sub-items such as tables, columns, and measures directly within their data items. This search functionality is available for users after they click on a specific item, ensuring quick access to relevant information without unnecessary navigation. New quick action for seamless navigation in the OneLake catalog To optimize workflows, you can now quickly open an item editor or viewer with a single click through a dedicated quick action instead of navigating to the Item Details page first. This improvement speeds up access to frequently used items and enhances productivity for all users. Purview Data Loss Prevention (DLP) policies for KQL and Mirrored DBs Security teams can use DLP policies to meet security and compliance requirements for sensitive data in the cloud. These policies leverage content scan to automatically detect the upload of sensitive information, and to trigger risk remediation actions (such as policy tips, audit logs and alerts) in semantic models and lakehouses. DLP policies support KQL DBs and Mirrored DBs (including Snowflake and Azure DBs). DLP coverage with this enhancement: KQL Database Mirrored Azure Cosmos DB Mirrored Azure DB for PostgreSQL Mirrored Azure SQL Database Mirrored Azure SQL Managed Instance Mirrored database Mirrored Snowflake Mirrored SQL Server Database Lakehouse (previously supported) Semantic model (previously supported) Get started with Data loss prevention policies for Fabric and Power BI to learn more. DLP policies restrict access action for lakehouses DLP Policies in Fabric help organizations detect sensitive information within their tabular data and surface it to end users and security administrators through policy tips, audit logs and alerts. The Restrict Access Action allows further control over data items once sensitive information has been discovered, by enabling security admins to define who can access the item upon DLP detection. This announcement means that once sensitive information is found within Fabric Lakehouse, unauthorized users will be blocked from accessing it until the data is removed. Items can be blocked from all users (excluding the data owners who always maintain access) or from guest users in the tenant. Learn more about Restrict Access in DLP in Fabric. Multi-tenant organization (MTO) (Generally Available) Support for multi-tenant organizations in Fabric is now generally available (GA). Entra ID users of type external member are supported across the Fabric platform. Users can authenticate, bring their own licenses from their home tenants and use Fabric workloads for development and consumption. There are some limitations when using Fabric with an external user. For more information refer to the Distribute Power BI content to external guest users with Microsoft Entra B2B documentation. https://youtu.be/qilDzEjPig4?si=qmQxTeaeBhjoPJIL OneLake OneLake security Managing granular data security across multiple applications and analytics engines is complex, often leading to either excessive restrictions or accidental exposure. That’s why we’re introducing OneLake security as a breakthrough in data protection. With OneLake security, you define access once, and Fabric enforces it consistently across all engines. Data owners can create security roles, grant precise permissions, and control access at the row and column level—for example, restricting Personally Identifiable Information (PII) while keeping other data available. This security propagates automatically, ensuring that whether users query via SQL or build Power BI reports, they only see what they’re authorized to access. OneLake security replaces the existing OneLake data access roles preview feature. Users start by creating OneLake security roles that grant access to specific data in a lakehouse. In addition to selecting tables and folders, OneLake security also allows for row and column level security to be defined. Using T-SQL, table access can be restricted to only specific rows where the T-SQL statement is true. To secure entire columns, roles can contain column level security definitions that block access to the sensitive columns. Assign members to your role to grant them access to only the allowed items in that role. With the role created, users can use any Fabric engine to query the data and see consistent results. Any queries through a Spark notebook are secured with OneLake security. The SQL Analytics Endpoint now uses the OneLake security definition to secure data when running in user’s identity mode. Semantic models can use Direct Lake mode to secure data using the security from OneLake. Even if users access the data in OneLake directly through API calls or OneLake file explorer, users are always restricted by the relevant OneLake security roles. OneLake security will be preview in the coming months, sign up for early access. Head over to OneLake security documentation for more information. External data sharing enhancements We have recently released several much-anticipated enhancements to the external data sharing feature. External data sharing allows in-place sharing of OneLake data across tenant boundaries. These updates include support for sharing multiple tables and folders, as well as entire Lakehouse schemas. Changes made to a shared lakehouse schema are automatically and immediately reflected in the consumer’s Lakehouse. Additionally, externally shared tables can now be consumed via the lakehouse’s SQL Analytics Endpoint and Semantic model, enabling seamless integration with Power BI reports. We have also expanded the types of data that can be shared to include KQL and SQL databases and introduced service principal support in the external data sharing APIs for automated management. For more details, check out the full announcement of external data sharing OneLake shared access signatures (SAS) (Generally Available) OneLake shared access signatures (SAS) are now generally available (GA)! OneLake SAS tokens provide secure-short-term, delegated access to your resources in OneLake, helping you share or distribute data through scoped-down SAS tokens. SAS tokens and user delegation keys are always backed by a Microsoft Entra identity and always limited to a 1-hour lifetime, ensuring that all access to your data is through an approved identity and for a limited period. You can learn more about how to expand your data estate with OneLake SAS in the OneLake documentation: What is a OneLake shared access signature. https://youtu.be/qilDzEjPig4?si=qmQxTeaeBhjoPJIL Data Engineering Write capabilities and PySpark support in Spark Connector for Fabric DW We are pleased to announce the addition of writing capabilities with the Fabric Spark connector for Fabric Data Warehouse (DW) in the Fabric Spark runtime. This connector utilizes a two-phase write process to a Fabric DW table. Initially, it stages the Spark dataframe data into intermediate storage, followed by the COPY INTO command to ingest the data into the Fabric DW table. This approach ensures scalability with increasing data volumes and supports multiple modes for writing data to a DW table. Additionally, we are excited to announce PySpark support for this connector. This means you no longer need to use a workaround to utilize this connector in PySpark, as it is now available as a native capability in PySpark. The connector will be included as a default library within the Fabric Runtime, eliminating the need for separate installation. To learn more about Spark Connector for Fabric Data Warehouse (DW), please refer to the documentation: Spark connector for Fabric Data Warehouse. Esri’s ArcGIS GeoAnalytics integration with Microsoft Fabric Spark (Preview) Esri is recognized as the global market leader in geographic information system (GIS) technology, location intelligence, and mapping, primarily through its flagship software, ArcGIS. Esri empowers businesses, governments, and communities to tackle the world's most pressing challenges through spatial analysis and location insight. We are pleased to share that Microsoft and Esri have partnered to bring spatial analytics into Microsoft Fabric and have launched public preview. Our collaboration with Esri introduces cutting-edge visual spatial analytics right within Microsoft Fabric Spark notebooks and Spark job definitions (across both Data Engineering and Data Science experiences). With its integrated product experience, it empowers Spark developers or data scientists to natively use ArcGIS capabilities to run GeoAnalytics functions and tools within Fabric Spark for transformation, enrichment, and pattern / trend analysis of data across different use cases without any need for separate installation and configuration. Example - How to transform the data with ArcGIS spatial function to uncover the pattern of interest, for instance summarizing the total number of policies of insured properties by hexagonal bins: Example - Understand the impact of natural hazards or current events on insured properties by bringing a dataset with probabilities of hurricane force winds and spatially joining it with insured properties. Spatial join links insured properties with wind speed probabilities, and with that for each property we would know the likelihood of hurricane force winds and can run predictive models to assess potential insurance claims. To learn more about this integration and capabilities, please refer to the documentation: ArcGIS GeoAnalytics for Microsoft Fabric (Preview). Deployment pipeline inside Spark Job Definition (Preview) The Spark Job Definition now supports the Deployment Pipeline. With this update, you can easily deploy your SJD (Spark Job Definition) item across different stages (Development, Testing, Production) and ensure that the proper state of the SJD item is synchronized across these stages. You can also customize the deployment with deployment rules to specify the default lakehouse and additional lakehouse of the SJD. Before triggering the deployment, you can verify the detail difference with the ‘Compare’ view. After the deployment is done, in the target stage/workspace, a new SJD item will be created based on the state from the source stage/workspace, and the association with other artifacts, such as Lakehouse and Environment, will also be set automatically. Deployment rule is supported to overwrite the default binding of default Lakehouse and Additional Lakehouse. By providing the Lakehouse ID, Lakehouse name, and the ID of the workspace where the Lakehouse is located, you can specify which Lakehouse should be set as the default in the target stage. You need to run the deployment after updating the deployment rule to make it effective. To learn more about this, please refer to the documentation: Spark Job Definition deployment pipeline support. Row-level and Column-level security in Spark We are pleased to announce the introduction of row and column level security for Spark within Microsoft Fabric. This update significantly enhances data governance by incorporating fine-grained security controls within Spark. Access control policies are established in OneLake security by specifying limiting factors for rows and columns in conjunction with tables during role definition. Spark uses these roles associated with the user executing the code and applies row and column data filtering accordingly before presenting the data to the user's code. These enhancements offer greater flexibility, stronger compliance, and simplified access management across Fabric’s unified data ecosystem. Introducing Pylance language support for Fabric Notebook Python developers using Fabric Notebook can now take advantage of Pylance, a powerful and feature-rich language server, to enhance their coding experience. With context-aware completions, better error detection, and improved code insights, Pylance makes PySpark and Python development smoother and more productive. Key Improvements with Pylance Smarter Auto-Completion moves beyond basic keyword and variable suggestions to context-aware completions, helping users quickly find relevant variable names and functions. Before Pylance With Pylance Enhanced Lambda Expression Support: More accurate completions within inline lambda functions, improving readability and efficiency for functional programming. Parameter Completions: Intelligent suggestions based on type hints and type inference, streamlining function calls. Improved Hover Information: More detailed insights when hovering over variables and code elements. Better Docstring Rendering: Clearer formatting and presentation of documentation strings for better readability. Error Markers & Semantic Highlighting: Improved error detection and code visualization, making debugging more intuitive. With Pylance in Fabric Notebook, writing Python and PySpark code is faster, more accurate, and more efficient. To learn more about Pylance in Notebook: Develop, execute, and manage Microsoft Fabric notebooks. Environment sharing across workspaces (Preview) You can now attach Environments from different workspaces in your Notebooks and Spark job definitions! This is made effortless with a brand-new explorer! Easily manage and utilize resources across multiple workspaces by exploring Environments from the workspaces you own, have access to, or that are shared with you by others. This feature provides flexibility in managing Environment permissions. Workspace viewers can use the Environment for running jobs without access to edit contents, while roles above workspace viewer can update the contents. To ensure only authorized users can access or update Environments, you can now manage Environments in one workspace, grant access to different users with different roles, or share the Environment with others with Read/Reshare/Edit permissions. Note that using an Environment from a different workspace does not break the compute and security configurations set by the admins. When you attach an environment from another workspace, both workspaces must have the same capacity and network security settings. Although you can select environments from workspaces with different capacities or network security settings, the session will fail to start. Furthermore, the compute configuration in that environment is ignored. Instead, the pool and compute configurations will default to the settings of your current workspace. To learn more about across attaching Environments: Create, configure, and use an environment in Fabric. Shortcuts now supported in Lakehouse Git metadata representation and in Fabric Deployment pipelines Git and Deployment Pipelines support for Lakehouse objects is a top ask across all our customer base, and we are excited to announce that now Shortcuts definitions under the Tables and Files section of lakehouse are supported in the Fabric Git/ALM platform. This is an exciting milestone, allowing customers to version and orchestrate Shortcuts using Fabrics Application Lifecycle Management capabilities. Now, Shortcuts are automatically exported as JSON metadata to the git repository connected to the workspaces. Also, you can modify Shortcut properties directly in git using your favourite authoring tool and import changes directly to the workspace. The Fabric Deployment pipelines work as expected, Shortcuts are now deployed across the stages defined in the pipeline configuration. This is the first step, on the upcoming releases, we will incrementally add support to additional object types under the Lakehouse, such as Folders, Tables, Views and more. Find out more information about the feature in the Lakehouse deployment pipelines and git integration documentation. Announcing Fabric User Data Functions (Preview) Fabric User Data Functions is a serverless platform that gives app developers and data engineers the ability to easily write and run applications on Fabric. User Data Functions empowers you to implement custom logic wherever you need to in your Fabric ecosystem by leveraging native integrations with Fabric data sources, Fabric Notebooks and Data pipelines. You can use your functions to perform data engineering tasks such as data validation or data cleaning, create integrations with external systems, or create re-usable function libraries. Learn more about this feature in the Fabric User Data Functions documentation. Link to YouTube video In this update, we added features that will help you make the best of your functions from the comfort of your browser. Portal editor You can now create, modify, delete or edit your functions directly in your browser. This experience gives you powerful tools to add to your functions code with the convenience of using the Fabric website portal. The editor features Intellisense and Pylance functionality to help you write quality Python code, as well as common editing functionality such as edit history, find and replace, and more. Insert code samples One of the most convenient features in the portal editor is the Insert Samples function that allows you to input code to quickly get started developing common use case patterns such as reading and writing to a Fabric data source, performing data transformations, and more. Add your favorite PyPI libraries! Another new feature is the Library management experience, which allows you to use the browser to add PyPI libraries into your project. Think of this as your requirements.txt file. You can select the library from a dropdown menu of names and choose a version that best suits your needs. The versions will be filtered to the ones compatible with the supported Python environment. New use cases and data sources! User Data Functions are deeply integrated with the Fabric ecosystem. You can now invoke your functions from different kinds of Fabric items such as Fabric notebooks, Power BI reports and Data pipelines. In addition to this, you can connect to Fabric data sources such as warehouses, lakehouses, SQL Databases, and Mirrored Databases for all your data applications. Learn more about this feature in the Fabric User Data Functions documentation. OPTIMIZE FAST and FSCK commands in Fabric Runtime 1.3 for Apache Spark (Generally Available) The Spark SQL FSCK REPAIR TABLE command and fast OPTIMIZE for V-Order are now available on Fabric Runtime 1.3 (Spark 3.5 / Delta 3.2). FSCK is designed to safely remove missing parquet files from the Delta transaction log, to restore table read consistency. This is not data recovery functionality, the missing parquet files and the data contained in it are lost. The command removes the references so the table can be back to a readable state. The command can also be run with a DRY RUN evaluation mode and will list all files that are missing in storage but still referenced by the Delta transaction log, to help you assess issues with the table before moving forward. Delta Lake’s OPTIMIZE VORDER can now be run with idempotency, meaning that previously V-Ordered parquet files that are already within the target file size won’t be considered for bin compaction. This significantly improves the performance of the OPTIMIZE command. Enable it by setting parquet.vorder.fast.optimize.enabled to true in the Spark session configuration directly on Notebooks, Spark Jobs or using Environments. Find more information in the Fabric Runtime 1.3 (GA) documentation. Fabric Spark Monitoring APIs (Preview) We’ve received valuable customer feedback emphasizing the need for API support to automate Spark job submission and monitoring. In response, we’re excited to introduce the preview of Fabric Spark Monitoring APIs—a robust set of tools designed to enhance observability and streamline the monitoring and management of Spark applications within Microsoft Fabric. To improve the developer experience, monitoring APIs for Fabric Spark applications are essential for optimizing performance, debugging issues, and ensuring efficient workload management. These APIs enable customers to automate Spark job management and monitor Spark jobs programmatically using APIs and SDKs. Key Capabilities of Fabric Spark Monitoring APIs With these APIs, users can: List all Spark applications within a workspace. Retrieve Spark applications for specific items, including Notebooks, Spark Job Definitions, and Lakehouse. Access detailed Spark application metrics using Livy ID. Leverage Spark History Server APIs to obtain execution metrics and job event details for a single Spark application, including jobs, stages, tasks, executors, and event logs. These capabilities empower users with greater automation, improved visibility, and deeper insights into Spark workloads within Fabric. Stay tuned for further enhancements as we continue refining these APIs based on customer feedback! Notebook Integration with User Data Functions (UDFs) (Preview) Introducing the preview of Notebook integration with User Data Functions (UDFs)! This new capability allows you to define custom logic and calculations that can be reused across multiple Notebooks, helping streamline workflows and enhance code modularity. With NotebookUtils, you can now seamlessly access and invoke UDFs directly from your Notebook code, making it easier than ever to integrate reusable functions into your data processing and analysis. Key Features & Scenarios Here’s how you can take advantage of UDFs within your Notebooks: Invoking a Function IntelliSense and autocomplete support function names, improving usability. To help you better understand UDF functions, we’ve introduced a help method display(myFunction.functionDetails). This provides a clear view of function details, including parameters and return types, ensuring that you can invoke functions correctly. Supported Languages This integration is available for Python, PySpark, Scala, and R, making it accessible across various data science and engineering workflows. For more details, check out our documentation: NotebookUtils (former MSSparkUtils) for Fabric. https://youtu.be/ktqKB4Bj1LQ?si=KPJ83e5D_jN8jbGF Data Science Copilot in Notebooks Agentic and UX enhancements The Copilot in Notebooks Agentic enhancement has introduced several significant improvements aimed at enhancing the user experience and productivity. One of the key enhancements is the enhanced conversation history, which allows users to maintain context and continuity in their interactions. Additionally, there have been chat and natural language output enhancements through enhanced algorithms that ensure more accurate and relevant code generation. The improved code generation capabilities now offer advanced reasoning for complex problem-solving, making it easier for users to write, debug, and understand code within the notebook environment. We’ve also added a new interaction modal for Copilot, on-cell and quick actions. The Copilot in Notebooks on-cell and quick actions introduces powerful features designed to streamline and enhance the coding workflow. The On-Cell Copilot Button, conveniently positioned above each notebook cell, allows users to perform advanced data manipulation tasks such as pivoting tables, joining datasets, and aggregating data based on specific criteria. Additionally, the Quick Actions Button, located just below the cell, simplifies tedious tasks using AI, such as fixing code errors and adding code comments. These features not only improve the efficiency of coding tasks but also enhance the overall user experience by providing intuitive and accessible tools directly within the notebook environment. With these enhancements, users can achieve more accurate and efficient results, making their coding process smoother and more productive. Fabric data agent Since the launch of AI skill in August 2024, we’ve improved conversational abilities, support for multiple data sources and so much more. To better reflect these enhanced agentic capabilities, AI skill is now Fabric data agent! Copilot and AI capabilities now available across all paid SKUs We’re thrilled to announce that we are removing the SKU requirement to access Copilot and AI capabilities across all paid SKUs. This means that customers on F2 and above will be able to use Copilot and AI features, such as Copilot in Power BI and Fabric data agent, to streamline workflows, generate insights, and drive impactful decisions. Fabric data agent integration with Azure AI Agent Service (Preview) We are excited to launch the integration of data agents in Fabric with Azure AI Agent Service from Azure AI Foundry. A critical component of Azure AI Agent Service is the ability to securely ground AI agent outputs in enterprise knowledge, ensuring responses are accurate, relevant, and contextually aware. Data agents in Fabric can retrieve knowledge using several specialized query language tools that help AI to generate SQL, KQL and DAX. By combining Fabric’s sophisticated data analysis over enterprise data with Azure AI Foundry’s cutting-edge GenAI technology, businesses can create custom conversational AI agents leveraging domain expertise. This seamless integration enables organizations to develop agents that are not only based on unstructured data in Azure AI Search or SharePoint but also integrate with structured and semantic data in Microsoft OneLake, thereby enhancing data-driven decision-making. Fabric data agent SDK (Preview) We are pleased to announce the preview of the Fabric data agent Python SDK. The Fabric data agent Python SDK library is a powerful tool designed to streamline the development and prototyping of AI assistants on the Fabric platform. It is intended for users who are looking to create, manage, and utilize Fabric data agents programmatically. The library provides a set of simple APIs that facilitate various operations, such as managing Fabric data agents and integrating various data sources for enhanced analysis and insights. It also makes it easier for users to interact with the Fabric data agent using the OpenAI Assistants API. This enables users to quickly prototype and experiment with Fabric data agents to refine their solutions. With the Fabric data agent Python SDK, users can automate workflows and reduce manual effort. Users can seamlessly create, update, and delete Fabric data agent artifacts, optimize resource configurations, and gain valuable insights from their data. To get started, users can leverage the comprehensive documentation and sample code provided with the SDK. By automating experimentation and validation processes, the Fabric data agent Python SDK ensures that developers can efficiently meet customer needs and deliver high-quality solutions, making it an invaluable tool for working with data agents. INSTALL the Fabric data agent. Data Warehouse AI functions in Data Warehouse AI functions are now available in private preview for Data Warehouse and Lakehouse SQL Endpoint, making it easier to bring AI-driven insights directly into your SQL workflows. With these built-in functions, you can summarize content, translate text, extract key data, analyze sentiment, and more - right from T-SQL. This eliminates the need for external processing, helping you streamline analysis and make faster, more informed decisions within your data warehouse. Figure 1: A few examples of AI functions usage. Refer to the detailed blog Functions in Data Warehouse to learn more and sign up for preview. Fabric User Data Functions in Data Warehouse Scalar Fabric functions are now available in private preview for Data Warehouse and Lakehouse SQL Endpoint, giving you the flexibility to extend SQL capabilities beyond built-in functions. With this feature, you can write custom functions in Python (and soon other languages) and invoke them directly through T-SQL, just like regular scalar user-defined functions. This allows you to bring complex logic, custom transformations, and advanced computations closer to your data, reducing the need for external processing. Refer to the detailed blog Functions in Data Warehouse to learn more and sign up for preview. Scalar SQL User-defined Functions Scalar SQL User-Defined Functions (UDFs) are now available in private preview for Data Warehouse and SQL analytics endpoint. Scalar SQL User-Defined Functions (UDFs) are a cornerstone of T-SQL programming, widely recognized and utilized for their ability to encapsulate business rules and calculations into a reusable code. This feature offers an efficient solution for promoting code modularity across T-SQL queries while natively leveraging Fabric Warehouse distributed engine. In an example below, by using four (4) different functions we can easily apply data masking logic on our customer table. Refer to the detailed blog Functions in Data Warehouse to learn more and sign up for preview. Developer Experiences in Fabric Warehouse IntelliSense for Collate Clause The COLLATE clause in Microsoft Fabric Warehouse and the SQL Analytics Endpoint of Lakehouse (LH) is essential for managing text-based data processing, ensuring accurate sorting, filtering, and comparisons. Given that Warehouse, SQL Analytics Endpoint of LH and other items support variations of case-insensitive and case-sensitive configurations, collation settings provide users with precise control over text handling in their workloads. By explicitly defining collation for VARCHAR and CHAR fields in table definitions, schema modifications, and queries, users can ensure consistency across transformations. The support for DATABASE_DEFAULT collation further simplifies schema management, allowing tables to inherit database-level settings for ease of administration and alignment with organizational standards. The collation feature in Microsoft Fabric Warehouse and SQL Analytics Endpoint is enhanced with IntelliSense and syntax highlighting, providing a more intuitive and efficient development experience. IntelliSense offers real-time suggestions, validation for collation names, helping users avoid syntax errors and ensuring compatibility with supported collation settings. Syntax highlighting further improves readability by visually distinguishing collation clauses, making it easier to identify and manage collation settings in CREATE TABLE, ALTER TABLE, SELECT, and CTAS statements. These features streamline query development, reduce errors, and enhance productivity when working with case-sensitive and case-insensitive data configurations across Microsoft Fabric's SQL environments. A few examples are: CREATE TABLE [SampleData_CI_UTF8] ( [SampleID] INT NOT NULL, -- Unique identifier for each sample [SampleValue] VARCHAR(50) COLLATE Latin1_General_100_CI_AS_KS_WS_SC_UTF8, -- Sample value with specified collation [CreatedAt] DATETIME2(6) NOT NULL -- Timestamp for when the sample was created ); CREATE TABLE [SampleData_DB_Default] ( [SampleID] INT NOT NULL, -- Unique identifier for each sample [SampleValue] VARCHAR(50) COLLATE DATABASE_DEFAULT, -- Sample value with specified collation [CreatedAt] DATETIME2(6) NOT NULL -- Timestamp for when the sample was created ); INSERT INTO [SampleData_CI_UTF8] ([SampleID], [SampleValue], [CreatedAt]) VALUES (1, 'Sample1', GETDATE()), -- Inserting sample data (2, 'Sample2', GETDATE()); INSERT INTO [SampleData_DB_Default] ([SampleID], [SampleValue], [CreatedAt]) VALUES (1, 'Sample1', GETDATE()), -- Inserting sample data (2, 'Sample2', GETDATE()); -- Collate in Select Select [SampleValue] COLLATE Latin1_General_100_CI_AS_KS_WS_SC_UTF8 from [SampleData_DB_Default] Select [SampleValue] COLLATE DATABASE_DEFAULT from [SampleData_CI_UTF8] --Collate in CTAS CREATE TABLE SampleDataCreate AS Select [SampleValue] COLLATE Latin1_General_100_CI_AS_KS_WS_SC_UTF8 as SampleValue_CI_UTF8 from [SampleData_CI_UTF8] CREATE TABLE SampleDataCreate AS Select [SampleValue] COLLATE DATABASE_DEFAULT as SampleValue_CI_UTF8 from [SampleData_CI_UTF8] -- Collate in ALTER Table Add New Column ALTER TABLE SampleData_CI_UTF8 ADD Column4 VARCHAR(10) COLLATE Latin1_General_100_CI_AS_KS_WS_SC_UTF8 NULL; ALTER TABLE SampleData_DB_Default ADD Column4 VARCHAR(10) COLLATE DATABASE_DEFAULT NULL; JSON data in OPENROWSET These IntelliSense and grammar updates make working with OPENROWSET, JSON more seamless and efficient. With improved syntax highlighting, and query validation, SQL development is now faster and more error-free. Live templates Writing T-SQL queries efficiently is crucial for database developers. Live Templates are predefined code snippets that can be inserted into your T-SQL editor with minimal effort. They help reduce repetitive coding, enforce best practices, and improve developer productivity. Key benefits: Faster query development Standardized SQL formatting Reduced errors in repetitive tasks Expand and collapse objects properly in filter and search We’re making search and filter experiences more intuitive! Now, objects that meet your search and filter criteria will automatically expand, giving you instant visibility into relevant data. Moving forward, we’ll expand only what’s required—keeping your object explorer clean and efficient unless no matches are found. Artifact Status Bar The Git item status bar component offers a comparable experience to the status bar in the workspace. When accessing the item page, you can view the details of the connection between the workspace and the Git repository, such as: The name of the branch to which the workspace is connected The time of the last sync event between the workspace and the repository A hyperlink to the most recent commit on the branch. Cancel query on closing editor Handling long-running queries efficiently is crucial for a seamless warehouse experience. To improve user control, we’re introducing an enhanced query cancellation prompt that ensures users can make informed decisions when closing the editor while a query is still executing. How it works 1. Prompting users when closing an active query If a user attempts to close the editor while a query is running, they will see a confirmation message: ‘Do you want to cancel the query?’ Yes: the query is canceled, and the editor closes immediately. No: the editor closes, but the query continues running in the Queries section, ensuring users don’t lose progress. This helps prevent accidental cancellations while still allowing users to exit the editor seamlessly. 2. Customizing future prompts When a user chooses Yes to cancel a query for the first time, they will see an additional prompt: ‘Do you want to see this message next time?’ Yes: the prompt will continue appearing for future query cancellations. No: the editor will automatically cancel queries without showing the confirmation message moving forward. This setting is user-specific, meaning each user can customize their experience individually. Users who opt out will no longer be interrupted, making their workflow faster and more efficient. Why this matters: Prevents accidental query cancellation – Ensures users don’t unintentionally stop important queries. Reduces interruptions – Users can choose whether they want to see the prompt in the future, keeping their workflow smooth. Personalized experience – Every user gets the flexibility to decide how they handle active queries when closing the editor. Show query editor shortcuts Navigating your data warehouse just got faster! Our keyboard shortcuts UI enhance efficiency across key areas: Object explorer – Quickly browse and manage database objects. Ribbon – Access essential commands with a single keystroke. T-SQL editor – Speed up query writing and execution. Results grid – Seamlessly filter, copy, and analyze query results. Why use keyboard shortcuts? Faster navigation – Reduce mouse dependency and move through objects quickly. Increased productivity – Execute queries, format code, and manage results seamlessly. Streamlined workflow – Spend less time on repetitive actions and more on data insights. SQL audit logs (Preview) We are excited to announce that SQL audit logs are now in preview in Microsoft Fabric Data Warehouse! Audit Logs provide a detailed record of warehouse activity, capturing essential information such as when events occur, which triggered them, and the T-SQL statement behind the event. This feature is crucial for security and compliance, helping organizations monitor access patterns, detect anomalies, and meet regulatory requirements. Previously, tracking warehouse events manual effort making security audits and forensic investigations cumbersome. With native audit logging in Fabric Data Warehouse, organizations gain automated, tamper-resistant logging, simplifying security operations. Whether you need to investigate unauthorized access, analyze query execution trends, or ensure adherence to governance policies, SQL Audit Logs provide the transparency and control needed to safeguard your data. Fabric Data Warehouse item permissions We are thrilled to announce the introduction of enhanced sharing capabilities in Microsoft Fabric Data Warehouse! With these new updates, you can now grant additional permissions to monitor queries and audit activities, providing deeper visibility into warehouse operations. These enhancements allow organizations to delegate access more effectively, enabling security teams, auditors, and operations personnel to track query performance, analyze workloads, and audit activity with the appropriate level of control. By improving security, governance, and operational insights, these new capabilities help organizations maintain compliance while ensuring efficient data management. What Permissions Can Be Assigned to Users? When it comes to assigning permissions, it's important to understand the different types of permissions available and their implications. Here are some core and custom permissions that can be assigned to users: Read: Allows users to view the data. Write: Grants users the ability to modify the data. Reshare: Enables users to share data with others. Monitor: Provides users with the ability to monitor database activities and kill sessions. Audit: Allows users to configure and access audit logs. Restore: Permits users to perform in-place restores of data. How to Assign Permissions Assigning permissions can be done through user interfaces on the share dialog: After you click on the option, we will be able to see the options surfaced on the dialog menu: You can also validate the permissions on the Manage Permissions option on the share menu: Item permissions are a fundamental aspect of data management, providing the necessary controls to secure, comply, and collaborate effectively. By understanding and implementing permissions like Monitor, Reshare, and Audit, organizations can enhance their data security posture and foster a collaborative environment. OneLake Security for Lakehouse Analytics SQL Endpoints Private Preview As data governance becomes more central, we're thrilled to introduce OneLake Security for SQL Analytics Endpoints, now available in Private Preview! This powerful capability simplifies how security is enforced in Microsoft Fabric by letting you configure access once in OneLake, and have that security respected across your SQL workloads. With this release, organizations can now govern data at scale with a consistent, unified approach—whether you're implementing centralized security controls or need granular SQL-based permissions. OneLake Security empowers teams to secure, simplify, and scale access across your Lakehouse architecture. Two Flexible Access Modes to Match Your Needs OneLake Security introduces two distinct access modes for SQL Analytics Endpoints: 1. User Identity Mode In this mode, the SQL Endpoint uses the signed-in user’s identity to access data in OneLake. It fully honors the RLS (Row Level Security), CLS (Column Level Security), and OLS (Object Level Security) rules defined in OneLake. Great for: Organizations that want centralized control and alignment with data lake-level security. 2. Delegated Identity Mode Here, the SQL Endpoint uses the workspace or artifact owner's identity to connect to OneLake. This enables traditional SQL-based security management with full support for GRANT, custom roles, masking, and other advanced database security features. Great for: SQL administrators and advanced use cases needing fine-grained SQL access control. User Identity x Delegated Mode Capability User Identity Mode Delegated Identity Mode Access Context Signed-in User Datawarehouse Owner OneLake RLS/CLS/OLS Enforced Not Enforced SQL GRANT on Tables Not Allowed Allowed SQL GRANT on Views/Procedures Allowed Allowed Dynamic Data Masking Not Supported Supported Custom SQL Roles Not Supported Supported With OneLake Security for SQL Endpoints, Microsoft Fabric continues its mission to make data governance intuitive and scalable. Whether you're building a self-service analytics culture or enforcing strict compliance policies, OneLake Security gives you the tools to do both—with confidence. Real-Time Intelligence Eventstream sources: MQTT, Solace, ADX, weather & Azure Event Grid Eventstream is a powerful feature in Fabric Real-time Intelligence that allows users to ingest, transform, and route real-time data streams to various destinations within Fabric. We are excited to announce the addition of five new sources and additional sample data streams. These new sources enhance Eventstream's streaming capabilities, enabling seamless data ingestion and real-time transformation across various data streams. Let’s dive into the functionalities of each connector and explore how they can benefit your data processing needs. MQTT connector: Connect to an MQTT broker, subscribe to specific topics, and stream data from those topics into Eventstream. Solace PubSub+: Read messages from a Solace PubSub+ Event Broker cluster and stream them into Eventstream for real-time data processing. Azure Data Explorer: Streams data from an Azure Data Explorer database in real-time into Fabric. Real-time Weather: Ingest live weather data for a selected city into Eventstream, including temperature, humidity, and wind speed. Azure Event Grid namespace: Stream MQTT, IoT, or any messages from Azure Event Grid namespace to Eventstream. Sample data streams: Kickstart your streaming projects with additional pre-built sample streams, including real-time bus tracking data and S&P 500 stock market data. These new connectors open a world of possibilities for data integration and analytics. To learn more about real-time streaming and processing in Fabric Eventstream, be sure to check out the Fabric Eventstream overview documentation. Can’t find your data sources? Let us know! Send us an email at [email protected] or fill out our survey. Eventstream CI/CD & REST APIs (Generally Available) Collaborating on data streaming solutions can be challenging, especially when multiple developers work on the same Eventstream item. Version control challenges, deployment inefficiencies, and conflicts often slow down development. Since introducing Fabric CI/CD tools for Eventstream last year, many customers have streamlined their workflows, ensuring better source control and seamless versioning. Now, we’re excited to announce the general availability (GA) of Eventstream CI/CD and REST APIs—making these capabilities even more accessible and powerful for all users. Key benefits of leveraging CI/CD tools in Eventstream: Enhanced Collaboration: With Git integration, developers can use GitHub or Azure DevOps to sync with the Fabric workspace and work in parallel on the same Eventstream item without conflicts. Streamlined Deployments: The Deployment pipeline feature accelerates and standardizes Eventstream deployments to various stages, such as testing and production workspace, with minimal manual effort in the Fabric UI. This ensures a more efficient and reliable deployment process. Automated Workflows: The availability of Eventstream REST APIs allows developers to build fully automated CI/CD pipelines and integrate external applications. This capability ensures quality, reliability, and productivity for data streaming projects, reducing manual intervention and potential errors. Increased Productivity: By leveraging these powerful CI/CD tools, teams can focus more on transformation within Eventstream and less on managing conflicts and deployment issues. This ultimately boosts overall productivity and project success. Overall, the GA of CI/CD and REST APIs for Fabric Eventstream empowers users to achieve a more efficient, reliable, and collaborative development experience. To learn more about Eventstream’s CI/CD, check out: Eventstream CI/CD - Git Integration and Deployment pipeline Eventstream REST API Expanded regional availability for Eventstream's managed private endpoints (Secure Outbound) Managed Private Endpoint (MPE) is a Fabric platform security feature that allows Fabric items, such as Eventstream, to securely connect to data sources behind firewalls or protected networks. Since we introduced this integration last year, many customers have relied on it to establish secure outbound connections between Eventstream and their data sources. This feature ensures that your data is transmitted securely over a private network, allowing you to fully harness the power of real-time streaming and high-performance data processing in Eventstream. The diagram below shows a typical setup using MPE in Eventstream. Managed Private Endpoints are now available in even more regions, making it easier for organizations worldwide to build secure, scalable streaming solutions. The table below lists supported regions for Eventstream’s MPE: To learn more about Managed Private Endpoints, check out the Connect to Azure resources securely using MPE in Eventstream. Connect to Eventstream using Microsoft Entra ID authentication We’re excited to introduce Microsoft Entra ID authentication for Eventstream’s Custom Endpoint! This feature enhances security by eliminating the need for SAS keys or connection strings, reducing the risk of unauthorized access. Instead, Entra ID authentication ensures that user permissions are directly tied to Fabric workspace access, allowing only authorized users to send and fetch data from Eventstream. The screenshot demonstrates how this feature works in Eventstream’s Custom Endpoint! Additionally, if you’re using an Azure resource like Azure Logic Apps with a system-assigned or user-managed identity, you can now assign Fabric workspace permissions to that identity. This enables Azure Logic Apps to seamlessly connect to Eventstream using Managed Identity authentication. The screenshot demonstrates how to enable identity in the Azure Logic Apps and assigning permission in the Fabric workspace. To learn more about Entra ID authentication in Eventstream’s Custom Endpoint, refer to our documentation Connect to Eventstream using Microsoft Entra ID authentication. Preview Real-Time Data Streams for Apache Kafka, Confluent Cloud, Amazon MSK & Amazon Kinesis Transforming data in Eventstream requires an actual schema derived from incoming data, which can slow down development and troubleshooting. To simplify this process, we are excited to introduce Data preview, a major usability enhancement for third-party connectors in Fabric Eventstream, including Apache Kafka, Confluent Cloud, Amazon Managed Streaming for Apache Kafka (MSK) and Amazon Kinesis Data Streams. With this new capability, users can preview a snapshot of their source data directly within Eventstream Edit mode and process data with inferred schemas. Why Data preview matters The Data preview feature allows users to: Enable Eventstream to infer the schema from incoming data, making it easier to configure operators such as filtering and aggregation. Verify if an Eventstream source is properly configured. Preview of real-time data snapshots to confirm data is ingesting as expected. How It Works Using Data preview in Eventstream is simple: Select a source connector in Eventstream (e.g., Confluent Cloud). Click on ‘Data Preview’ tab to view a snapshot of the source data. Change and match the source data format for preview. The screenshot below shows a snapshot of the Confluent data streams in Eventstream Edit mode: With Data preview, teams can build, test, and deploy Eventstream items faster and with greater confidence. Get started today and experience the power of real-time processing for your third-party connectors in Eventstream! Continuous improvements to Eventhouse Get Data Experience There are a few different methods to get data into the Real-Time Intelligence workload, depending on your organization’s needs. Data can be pushed or pulled into an Eventstream from one of the many connectors and then landed in an eventhouse. Alternatively, there are several ways to directly ingest data into Eventhouse. Data can be directly from: Local files Azure storage Amazon S3 Event hub Eventstreams OneLake Get data in Eventhouse offers a step-by-step process, guiding you from importing the data, through inspecting the incoming data, creating or editing the destination table schema to exploration of the ingested result. Over the past few months, our team has been working tirelessly to bring new features to the Get Data wizard, creating a simpler interface, quicker navigation, and added automation, all aimed at delivering a better user experience and improved performance. The main changes introduced: Automated schema optimization: Since the Eventhouse engine is highly optimized for datetime and string operations, in certain cases mapping imported data to these data types can offer a significant boost in query performance. By introducing usage of the inferred schema plugin, in most applicable cases an optimal data mapping will be inferred and automatically applied to your data, enhancing Eventhouse’s query and indexing performance. For instance, it can detect columns storing Unix date-time values as long and convert them to datetime. Similarly, it might recognize that a column named ‘id’ using a long type should be converted to string. Any such automatic mapping changes are clearly reflected in the schema editor, with a lightbulb icon and a short explanation of the mapping applied. Changes can easily be manually reverted using the ‘Type’ dropdown menu, although this is usually not recommended. Simplified schema inspection: The Schema Inspection step allows users to preview the destination table schemas, modify the schema if any change is needed or extract the KQL commands for table and schema creation. Based on the feedback received from our customers, we have improved the schema Inspection experience to make schema preview and edit a seamless experience. Users can now easily switch between the schema preview; command viewer and schema edit modes with an intuitive switcher experience. We have also simplified the inference file, formats, mapping and nested JSON options to make them more accessible. 2. Real-Time data sampling: We have graduated Sample data option in the inspect step to real-time data sampling. This allows users to preview how the data would look like when ingested, even before finalizing the data schema. 3. Automatic detection of header row for CSV files: The Get Data wizard now seamlessly detects if a CSV has a header row and uses it for column names. The column data type is inferred based on the data in the remaining rows. This makes the process of schema definition, when your file has headers a painless process. Build event-driven workflows with Azure and Fabric Events (Generally Available) Azure and Fabric Events, a powerful capability that allows organizations to capture, process, and respond to events across Microsoft Fabric is now generally available. With these events, businesses can integrate event-driven solutions into their workflows, enabling seamless automation, enhanced observability, and faster decision-making. What are Azure and Fabric Events? Azure and Fabric Events offer a capability within Real-Time Intelligence that enables you to: Ingest events that are available in Microsoft Fabric like Onelake events, Azure blob storage events, Job events,Workspace item events Filter those events using rich filtering capabilities on event schema properties. Integrate those events to consumers in Microsoft Fabric like Activator for setting event-based alerts or Eventstream to stream events to other destinations. With Azure and Fabric Events, organizations can reduce latency, improve operational efficiency, and build scalable event-driven applications. To learn more, please go to Azure and Fabric Events documentation and for the full announcement, refer to the announcement blog. Eventhouse OneLake availability now supports backfill Eventhouse OneLake availability allows creating a delta parquet representation of data in Eventhouse. Previously, when you turned availability ON, only new data was made available in OneLake, with no backfill of existing data. This could cause inconsistencies between the data in Eventhouse and OneLake. Now, OneLake Availability supports backfill, making all existing and new data in Eventhouse available, regardless of when you turn it ON. This is the default behavior when you enable availability via the UI. Learn more about Eventhouse OneLake availability. Improved Activator alerts from Power BI We’ve made it easier than ever to create and manage Activator alerts on your Power BI reports. We’ve redesigned the Power BI ‘Set Alert’ experience so that you can conveniently manage your alerts entirely within your reports, without having to open Activator. We’ve also streamlined the experience so that you can set up an alert with fewer steps. To check out the new experience, open a Power BI report and select ‘Add Alert’ on a visual, or choose ‘Set Alert’ from the ribbon. Figure 2: The improved ‘Set Alert’ experience in Power BI makes it easier than ever to create and manage Activator alerts on your reports. End-to-end Real-Time Intelligence samples We are excited to announce a brand new RTI sample experience which allows you to create a fully working end-to-end RTI flow within seconds. The sample flow allows you to explore the main features of Real-Time Intelligence with sample data. It provides a comprehensive end-to-end solution, demonstrating how Real-Time Intelligence components work together to stream, analyse, and visualize real-time data in a real-world context. You can access the samples from the RTI workload home page. Select the sample scenario of your choice. Choice of Bike rentals or S&P 500 Stocks data. Create a sample solution with all RTI items within seconds. Learn more about End-to-end sample. Synapse Data Explorer to Eventhouse migration tooling (Preview) The next generation of Azure Synapse Data Explorer offering is evolving to become Eventhouse, part of Real-Time Intelligence in Microsoft Fabric. For customers looking to migrate to Eventhouse, we are providing a migration tooling that allows you to seamlessly migrate Synapse Data Explorer cluster to an Eventhouse in Fabric. The migration process is performed using Fabric REST API endpoints. The recommended steps for performing the migration are as follows: Validate: Use the Validate migration to Eventhouse endpoint to check whether the Azure Synapse Analytics Data Explorer cluster can be migrated to an eventhouse. Migrate: Use the Migrate to Eventhouse with the migrationSourceClusterUrl payload to create an eventhouse with the migration source cluster URL. The process runs asynchronously to create a new eventhouse and migrate all databases from the source cluster to the eventhouse. Monitor: Use the Monitor migration progress to track the progress of your migration. Verify: Verify the migration by checking the eventhouse state is Running, and that the migrated databases appear in the KQL database list. To learn more about how the API endpoints can be called directly or in an automated PowerShell script refer to our migration tool documentation. https://youtu.be/pluk-b8XVj4?si=WYcsvCJG4SIhzaHf Data Factory Enterprise readiness VNET Gateway support for Data pipelines Support for data pipeline functionality on the VNet data gateway is now available in preview. The VNet data gateway facilitates connections to data sources that are either behind firewalls or accessible within your virtual network. This feature enables the execution of data pipeline activities on the VNet data gateway, ensuring secure connections to data sources within the VNet. Unlike on-premises data gateways, VNet data gateways are managed by Microsoft, consistently updated, support auto-scaling, and deactivate when not in use, making it cost-effective. To learn more, refer to the documentation: What is a virtual network (VNet) data gateway? Best-in-class connectivity and enterprise data movement In the fast-evolving data integration landscape, Data Factory continues to enhance the existing connectors to provide a seamless, high-performance experience. With a focus on improving connector efficiency and expanding capabilities, recent updates have made significant advancements to Salesforce and Lakehouse connectors. These improvements not only boost performance but also enable more sophisticated data handling, ensuring that enterprises can extract, transform, and load data with greater accuracy and efficiency. Performance improvement in Salesforce connector in data pipelines Salesforce is a critical data source for many organizations, housing valuable customers and business data. To enhance data movement efficiency, Data Factory has introduced performance optimization in the Salesforce connector for pipelines. Optimization allows you to fetch the data concurrently from Salesforce by leveraging the parallelism capability, thus significantly reducing extraction times for large datasets. Lakehouse connector now supports deletion vector and column mapping for delta tables in data pipelines The Lakehouse connector in Data Factory has been upgraded to provide deeper integration with delta table. Two major new capabilities enhance data processing workflows: 1. Support for deletion of vector Delta table uses deletion vectors to track deleted records efficiently without physically removing them from storage. With this new feature in the Lakehouse connector, users can: Read Delta tables while respecting deletion vector, ensuring that deleted records are automatically excluded from queries. Improve performance by leveraging soft deletions instead of physical file modifications, making data updates and maintenance more efficient. Enable compliance with data retention policies by retaining historical data for auditability while ensuring deleted records are filtered out from active queries. 2. Column mapping support for delta tables Delta table's column mapping capability allows for more flexible schema evolution, ensuring that changes in table structure do not disrupt data workflows. With column mapping support in the Lakehouse connector, users can: Read from an existing delta Lake table with column mapping name/id mode enabled Write to existing delta lake table with column mapping name/id mode enabled Auto-create table with column mapping name mode enabled when sink table does not exist and source dataset columns contain special chars & whitespaces. Auto-create table with column mapping name mode enabled when table action is overwriting schema and source dataset columns contain special chars & whitespaces. These enhancements ensure that data engineers can work with delta tables more efficiently, improving data governance, performance, and maintainability. To learn more about how to Configure Lakehouse in a copy activity refer to our documentation. New and updated Certified Connectors for Power BI and Dataflows As a developer and data source owner, you can create connectors using the Power Query SDK and have them certify through the Data Factory Connector Certification Program. Certifying a Data Factory connector makes the connector available publicly, out-of-box, Microsoft Fabric Data Factory and Microsoft Power BI in the following experiences This month we are happy to list the newly updated certified connectors that are part of the Microsoft Data Factory Connector Certification Program. Be sure to check the documentation for each of these connectors so you can see what’s new with each of them. New connectors ADP Analytics Dynatrace Grail DQL Updated connectors Anaplan Asana BQE Core BuildingConnected Delta Sharing SolarWinds Service Desk Supermetrics Windsor Worksplace Analytics Zendesk Data Simplifying Data Ingestion with Copy Job Copy Job is making data ingestion simpler, faster, and more intuitive than ever and is now generally available. Whether you need batch or incremental data movement, Copy Job provides the flexibility to meet your needs while ensuring a seamless experience. Since its preview last September, Copy Job has rapidly evolved with several powerful enhancements. Let’s dive into what’s new! Public API & CICD support Fabric Data Factory now offers a robust Public API to automate and manage Copy Job efficiently. Plus, with Git Integration and Deployment pipelines, you can leverage your own Git repositories in Azure DevOps or GitHub and seamlessly deploy Copy Job with Fabric’s built-in CI/CD workflows. VNET gateway support Copy Job now supports the VNet data gateway in Preview! The VNet data gateway enables secure connections to data sources within your virtual network or behind firewalls. With this new capability, you can now execute Copy Job directly on the VNet data gateway, ensuring seamless and secure data movement. Upsert to Azure SQL Database & overwrite to Fabric Lakehouse By default, Copy Job appends data to ensure no changed data is lost. But now, you can also choose to upsert data directly into Azure SQL DB or SQL Server and overwrite data in Fabric Lakehouse tables. These options give you greater flexibility to tailor data ingestion to your specific needs. Enhanced usability & monitoring We’ve made Copy Job even more intuitive based on your feedback, with the following enhancements: Column mapping for simple data modification to storage as destination store. Data preview to help select the right incremental column. Search functionality to quickly find tables or columns. Real-time monitoring with an in-progress view of running Copy Jobs. Customizable update methods & schedules before job creation. More connectors, more possibilities! More source connections are now available, giving you greater flexibility for data ingestion with Copy Job. And we’re not stopping here—even more connectors are coming soon! 25_Feature_Summary What’s next? We’re committed to continuously improving Copy Job to make data ingestion simpler, smarter, and faster. Stay tuned for even more enhancements! Learn more about Copy Job in: What is Copy job in Data Factory Mirroring Mirroring for Azure SQL Database protected by a firewall (Preview) You now can mirror Azure SQL Databases protected by a firewall. Using either the VNet data gateway or the on-premises data gateway for mirroring is available. The data gateway facilitates secure connections to your source databases through a private endpoint or from a specific private network. Learn more about Mirroring for Azure SQL Database from Microsoft Fabric Mirrored Databases from Azure SQL Database. Mirroring for Azure Database for PostgreSQL Flexible Server (Preview) Database Mirroring now supports replication of your Azure Database for PostgreSQL Flexible Server into Fabric! Now you can continuously replicate data in near real-time from your Flexible Server instance to Fabric OneLake. This enables seamless data integration, allowing you to leverage Fabric’s analytics capabilities while ensuring your PostgreSQL data remains up to date. By mirroring your PostgreSQL data into Fabric, you can enhance reporting, analytics, and machine learning workflows without disrupting your operational database. To learn more, please reference the PostgreSQL mirroring preview blog. Open Mirroring UX improvements We’ve made improvements to our end-to-end in-product experience for Open Mirroring. With these changes, you can now create a Mirror DB and start uploading or dragging and dropping parquet and CSV files. It’s now easier than ever to get started with building your own Open Mirror source and allow you to test our replication technology before productionizing with APIs. Once your files are uploaded, you can also upload changes and updates to the data with the __rowMarker__ field specified to our change data capabilities. Transformations Save a new Dataflow Gen2 with CI/CD support from a Dataflow Gen1, Gen2, or Gen2 (CI/CD) Customers often would like to recreate an existing dataflow as a new dataflow Gen2 (CI/CD), getting all the benefits of the new GIT and CI/CD integration capabilities. Today, to accomplish this, they need to create the new Dataflow Gen2 (CI/CD) item from scratch and copy-paste their existing queries or leverage the Export/Import Power Query template capabilities. This, however, is not only inconvenient due to unnecessary steps, but it also does not carry over additional dataflow settings. Dataflows in Microsoft Fabric now includes a ‘Save as’ feature in preview, that in a single click lets you save an existing dataflow Gen1, Gen2 or Gen2 (CI/CD) as a new Dataflow Gen2 (CI/CD) item. Incremental Refresh for Dataflow Gen2 and new support to Lakehouse as destination with incremental refresh (Generally available) Incremental Refresh for Dataflow Gen2 is now generally available! Incremental Refresh for Dataflow Gen2 allows you to refresh only the buckets of data that have changed, rather than reloading the entire dataset on every dataflow refresh. This not only saves time but also reduces resource consumption, making your data operations more efficient and cost-effective. These new capabilities are designed to help you to be successful with your data integration needs and be as efficient as possible. Try it out today in your fabric workspace! Learn more about Incremental Refresh in Dataflow Gen2: Incremental refresh in Dataflow Gen2. Check ongoing validation status of a Dataflow Gen2 with CI/CD support When you click Save & run in Dataflow Gen2 with CI/CD support, the process that gets triggered is two-fold: Validation: it’s a background process where your Dataflow gets validated against a set of rules. If it passes all validations and no errors are returned, then it’ll be successfully saved. Run: Using the latest published version of the Dataflow, a refresh job gets triggered to run the Dataflow. If you only wish to trigger the validation process, you only need to click the ‘Save’ button. What if you want to check the status of the validation? You now have a new entry point in the home tab of the ribbon called Check validation which you can click at any time to give you information of the ongoing validation or the result of a previous validation run. Be sure to give this a try whenever you want to check the results of a save validation. Orchestration Apache Airflow Job (Generally Available) The Apache Airflow job in Microsoft Fabric is now generally available, providing a fully integrated Apache Airflow runtime for developing, scheduling, and monitoring Python-based data workflows using Directed Acyclic Graphs (DAGs). What's New: Introducing Fabric runtime versioning for Apache Airflow job – This includes Fabric runtime version 1.0, which comes with Apache Airflow 2.10.4 and Python 3 as the default runtime. Public API – APIs are now available to interact with Apache Airflow jobs for seamless management. Git Integration & Deployment pipeline support – Users can utilize their Git repositories (Azure DevOps/GitHub) and deploy with Fabric’s built-in CI/CD workflows. Diagnostic logs – Users can access Apache Airflow generated logs through the Apache Airflow job UI for enhanced observability. Learn more about Apache Airflow job in Microsoft Fabric in What is Apache Airflow job? OneLake file triggers for pipelines The Fabric Data Factory team is thrilled to announce that the pipeline trigger experience is now generally available (GA) and now includes access to files in OneLake! This exciting new improvement to pipeline triggers in Fabric Data Factory means that you can now automatically invoke your pipeline when files or folders have files that arrive, delete, or rename! We’ve previously supported Azure blob file events in Fabric Data Factory like ADF & Synapse but now that Fabric users are leveraging OneLake as the primary data hub, we’re excited to see the pipeline patterns that you’ll build using OneLake file triggers! Variable libraries for pipelines (Preview) One of the most requested features in Fabric Data Factory has been support for modifying values when deploying workspace changes between environments using Fabric CICD. To accommodate this, ask, we have integrated pipelines into the new Fabric platform feature called Variable Libraries. With Variable Libraries, you can assign variables to unique values based on different environments, i.e. dev, test, prod. Then when you promote your factory to high environments, you can use different values from the library providing the ability to change values when pipelines are promoted to new environments. This new preview feature will be super useful not just for CICD but also generically allows you to replace hardcoded values with variables anywhere in your pipelines to achieve the same functionality as global parameters in Azure Data Factory as well. Spark Job Definition pipeline activity parameter support The Spark Job Definition (SJD) activity in Data Factory allows you to create connections to your Spark Job Definitions and run them from your data pipeline. And we are excited to announce that parameterization is now supported in this activity! You will find this update in the Advanced settings where you can configure your SJD parameters and run your Spark Job Definitions with the parameter values that you set, allowing you to override your SJD artifact configurations. Azure Databricks jobs activity now supports parameters Parameterizing data pipelines to support generic reusable pipeline models is extremely common in the big data analytics world. Fabric Data Factory provides end-to-end support for these patterns and is now extending this capability to the Azure Databricks pipeline orchestration activity. Now when you select ‘Jobs’ as the source of your ADB action, you can send parameters to your ADF job allowing maximum flexibility and power of your orchestration jobs. User data functions in Data pipelines (Preview) User Data Functions are now available in preview within Data pipeline's Functions activity. This new feature is designed to enhance your data processing capabilities by allowing you to create and manage custom functions tailored to your specific needs. Key Highlights Custom functionality: User Data Functions enable you to define custom logic and calculations that can be reused across multiple Data pipelines. This allows for more flexible and efficient data processing. Integration in data pipelines: You can add User Data Functions as activities within your Data pipelines. This is done by selecting the Functions activity in the pipeline editor, choosing your User Data Functions as the type, and providing any necessary input parameters. Check out our documentation to learn more about how to User Data Functions in your data pipelines. Data Factory pipelines now support up to 120 activities We’ve increased the default activity limit from 80 activities to 120 activities! You can now utilize an additional 40 activities to build more complex pipelines for better error handling, branching, and other control flow capabilities. https://youtu.be/spe7ZMImHH0?si=uf7NmQ6CNp2iDJLO Dataflow Gen2 Dataflow Gen2 with CI/CD capabilities You can now add Dataflow refresh activities to your pipelines in Fabric Data Factory that include the new version of Dataflows: Check out our documentation on Dataflow Gen2 with CI/CD and Git integration to learn more. Data pipelines Data pipelines have supported CI/CD capabilities and REST APIs support is now generally available. The team just added Service Principal Name (SPN), and Variable libraries support for Data pipelines. Check out our documentation on CI/CD for Data pipelines and REST API capabilities for Data pipelines to learn more. Mounting ADF (Preview) CI/CD and REST APIs support is now available for the Azure Data Factory item (Mounting ADF). Mirrored database (Generally Available) The mirrored database’s CI/CD support is now Generally Available. Learn more from CI/CD for mirrored databases. The REST APIs support has been Generally Available including the SPN support. Check out our documentation on Mirroring Public REST APIs. Copy Job (Generally Available) The Copy job item’s CI/CD and APIs support is now Generally Availability. This includes SPN support for Copy job. Check out our documentation on CI/CD for Copy job to learn more. Parameterization Parameterized connections in Data pipelines Enhancing your Data Integration experience What are Parameterized Connections? Parameterization of data connections in Data pipelines allows you to specify values for connection placeholders dynamically. This means you can pre-create data connections for various sources, such as Azure Blob Storage, SQL Server or any other data source supported by data pipelines, and reference them through data pipeline’s dynamic expressions at runtime. This feature empowers you to create more flexible and adaptable data pipelines, capable of connecting to different instances of data connections of the same type, such as SQL Server, without altering the pipeline definition. Key benefits: Flexibility: Use the same data pipeline definition to dynamically connect to various instances of data connections. Efficiency: Minimize the need for multiple pipeline definitions, reducing complexity and maintenance effort. Scalability: Easily manage and scale your data integration processes by leveraging dynamic expressions to handle connection values. How it works: During the pipeline run, dynamic expressions within the data pipelines specify values for the connection placeholders, enabling seamless integration with pre-created data connections. This innovation ensures that your data pipelines are not only more efficient but also highly customizable to meet your specific requirements. We believe this new feature will significantly enhance your data processing capabilities and streamline your workflows. We can't wait for you to experience the benefits of parameterized connections in your data integration projects. Table Name parameter support for data destinations In Dataflow Gen2, you can create parameters, they serve as a way to easily store and manage a value that can be reused throughout your Dataflow. Major feedback that we’ve heard from our users is the lack of this capability in the Data destination experience for Dataflow Gen2. Thanks to the feedback, we’re now introducing the first support for parameters in the data destination experience where you can set a parameter to be used for the Table name of your destination. This is available to all destinations that support this field and we’re working on extending this support to other areas of the data destination experience. Try out this new capability and let us know what you think. AI-powered experiences Efficiently build and maintain your Data pipelines with enhanced capabilities for Copilot in Data Factory In November 2024, we announced the preview of 3 innovative capabilities in Copilot for Data Factory (Data pipeline). Today, we are excited to make these features generally available, with enhancements to make your data integration even more efficient and effortless. Check out the blog post on Efficiently build and maintain your Data pipelines with Copilot for Data Factory: new capabilities and experiences to learn more. Effortlessly generate your data pipelines: Understand your business intent and effortlessly translate it into data pipeline activities to build your data integration solutions. In the enhanced capability of Copilot, we can easily build more complex Data pipeline activities e.g. switch activity, metadata driven pipeline, etc. You can also update your pipeline settings and configurations in batches with multiple activities! Efficiently troubleshoot error messages in your data pipeline Copilot. Diagnose and resolve pipeline errors more intuitively by providing clear and actionable summary. Easily understand your complex data pipelines: Understand your complex pipeline configurations effortlessly by getting a clear and intuitive summary provided by Copilot. Data integration shared experiences Partner workloads We are thrilled to share the significant updates and newly released workloads for this month, showcasing the incredible efforts and innovative work by our amazing partners. Workloads (Generally Available) Osmos AI Data Wrangler Automate Data Ingestion with Osmos AI Data Wrangler for Microsoft Fabric As enterprises scale AI adoption, they must ensure all their data is AI-ready. However, enterprise data is often messy - semi-structured or unstructured, arriving in inconsistent formats from customers, partners, suppliers, and internal systems. Traditional ETL pipelines require constant engineering effort to adapt to schema drift, missing fields, and poor data quality, slowing down AI and analytics initiatives. Osmos AI Data Wrangler enables autonomous data transformation as a Workload on Microsoft Fabric. Osmos’ agentic AI automates data ingestion by intelligently cleaning, transforming, and validating your data, seamlessly normalizing messy bronze data into silver tables in your Lakehouse. Get it now: Osmos AI Data Wrangler for Microsoft Fabric Osmos AI Data Wrangler is now generally available! Osmos’ AI Data Wrangler is now generally available (GA), featuring self-configuration capabilities, featuring self-configuring Wrangler Context. This feature allows Wranglers to understand your business rules and apply it to data transformations. With the new Wrangler Context feature, businesses can auto-configure their Wranglers using existing documentation and code snippets, giving you easy-to-configure high performance data wranglers quickly. Why enterprises choose Osmos AI for data ingestion & transformation Osmos helps businesses across retail, manufacturing, finance, and audit unlock millions in savings while accelerating insights and new market opportunities. ✔ Unify clean data into Lakehouse - Normalize disparate data coming from multiple internal and external sources into clean, actionable SQL-ready data. ✔ Improve data quality - AI-powered validation ensures structured, clean, and accurate data for analytics and decision-making. ✔ Streamline workflows - Eliminate manual data cleanup by enabling Wranglers to autonomously learn from documentation and business rules. ✔ Enhance AI & analytics readiness - Deliver trusted, structured data ready for AI-driven insights and enterprise analytics. Check out the Osmos AI Data Wrangler with Wrangler Context video. Power BI Designer (Generally Available) Microsoft Fabric in collaboration with PowerBI.tips, are excited to share that Power BI Designer is now generally available! You won’t want to miss out on this time saving application. Say goodbye to bland, cookie-cutter reports and hello to dazzling, highly stylized masterpieces. What’s Power Designer All About? Power Designer is sleek, intuitive, and fun, making designing reports feel less like work and more like unleashing your inner artist. It’s packed with features that’ll have you saying, ‘Why didn’t I have this sooner?’ Get it now: Power Designer Workload. Let’s dive into the magic: Craft themes like a pro: create detailed theme files for your Power BI reports with ease. Customize colors, fonts, and styles to match your brand. Real-Time visual vibes: watch your Power BI visuals update live as you build your style. Multi-page: Add background images to each page with a snap, transforming your reports into polished, magazine-worthy layouts. AI-Powered: Let AI take the wheel with auto-placement of visuals in your multipage templates. Preview: Test your shiny new theme on reports already published in your workspaces with the preview feature. Now that Power Designer has officially been released, it’s time to jump in and start creating. Head to your Fabric Workspaces, fire up Power Designer, and let your imagination run wild. Ready, set, design! Let’s make some report magic happen! Learn more at PowerBI.tips Designer YouTube video: Introducing Power Designer: Unleash Your Inner Report Wizard! Newly released workloads Profisee MDM Workload Profisee has introduced the first native Master Data Management (MDM) workload within Microsoft Fabric, seamlessly integrating its MDM platform into Fabric’s environment. This breakthrough allows users to manage and unify enterprise data without leaving the familiar Fabric interface. Get it now: Profisee MDM Workload for Microsoft Fabric. By embedding MDM capabilities directly into Microsoft Fabric, Profisee empowers organizations to: Ensure data consistency – Align and integrate data from multiple sources while enforcing standardized data governance within Fabric’s OneLake. Enhance data quality – Leverage intelligent matching, merging, and standardization to create trusted ‘golden records’ for your most critical business data. Streamline workflows – Manage data within Fabric itself, eliminating the need for external tools and reducing context switching. Accelerate AI & analytics initiatives – Deliver high-quality, consumable data ready to fuel enterprise applications of generative AI and advanced analytics. This integration marks a major advancement in data management, offering a unified platform for data stewardship, modeling, and governance. The native MDM experience in Fabric simplifies the transformation of raw data into actionable insights— driving business outcomes faster than ever possible before. For organizations leveraging AI and advanced analytics, having trusted, consumable (aka ‘gold medallion’) data is critical. Profisee’s deep integration with Microsoft Fabric ensures businesses can confidently rely on their data to make informed decisions and drive innovation. This collaboration between Profisee and Microsoft represents a significant leap forward, enabling enterprises across industries to unlock insights, fuel opportunities, and finally bring their data into the age of AI to become data-driven at scale. YouTube Video – Welcome to Profisee's featured workload in Microsoft Fabric Figure: Profisee enables your medallion architecture to deliver consumable, trusted data for AI and analytics. Figure: Profisee can match, merge and standardize data from different sources. Lumel PowerTables Workload The Lumel Fabric workload is now in preview. PowerTable allows business users to build no-code, writeback-enabled table apps on Microsoft Fabric. It connects LIVE (with bi-directional sync) to your cloud data warehouse tables in platforms such as Fabric SQL, Fabric Data Warehouse, Azure SQL, Databricks, Snowflake, Amazon Redshift, Google BigQuery, and PostgreSQL. Get it now: Lumel PowerTables Workload. There are 3 key use cases for PowerTable: Build tabular apps for data typically maintained in Excel such as product price lists, contract trackers, project status updates, etc. Manage master data / reference data / meta data supporting your reporting and planning applications. PowerTable supports forward-looking master data (unlike your ERP and MDM platforms) such as prospective customers or products that are not yet launched. Build applications on top of your semantic models and facilitate user data input and data writeback. PowerTable delivers the following features: Bulk insert and bulk edit records Cell-level commenting and collaboration Change log with audit trail Support for slowly changing dimensions Row-level CRUD user permissions Field-level permissions Sequential, multi-level approvals Rule-based approvals Outlook and Teams notifications Workflow automation Triggers and cascading updates Webhook integration … and more Unlike products like Airtable or Smartsheet, which typically struggle to handle more than tens or hundreds of thousands of rows, PowerTable is highly scalable and can support millions of rows. This is possible because our architecture separates the user interface from storage and compute and uses pushdown SQL statements to perform all processing on the underlying database of your choice. Visit our website www.lumel.com to learn more. PowerTables Introduction in action video. SAS Decision Builder Workload (Preview) Announcing SAS Decision Builder in preview, a new workload from longtime Microsoft partner and analytics vendor SAS. This powerful SaaS solution is designed to help organizations automate, optimize, and scale their decision-making processes in real-time, whether they are managing complex business rules, integrating machine learning models, or using Python code. Get it now: SAS Decision Builder workload. There are numerous use cases across many industries, including financial services (loan approvals, financial products), manufacturing (product quality), and public sector (fraud identification, help with choosing a government service). Decision Flow in SAS Decision Builder Key features: Construct business rules and decisioning flows: Access and process data from all your sources to create decisions that align with business goals. Robust governance capabilities: Test, validate, schedule, run and monitor decisions within the unified Fabric environment. Adjust decisions as business needs evolve. Add value to your ML Models: Call your ML models within your Decision Flow to further enhance your decisions. Integrate Python code files into decisions: Bring advanced logic and flexibility into your automated processes. Partner workloads - SAS Video Demo Getting started: Access SAS Decision Builder from the Workloads tab within your Microsoft Fabric instance. Select the SAS Decision Builder Workload Hub page, select ‘Add Workload,’ and transact through the Azure Marketplace process. Once complete, you can start building your decisions with SAS Decision Builder. Striim SQL2 Fabric Workload Microsoft Fabric’s robust ecosystem empowers partners to integrate custom capabilities through the Fabric Workload Development Kit. Striim proudly introduces a Fabric workload designed for real-time data ingestion, processing, and AI-driven analytics—delivering seamless integration, scalability, and actionable insights. The Striim workload builds on the Open Mirroring capabilities provided by Fabric to provide users a managed copy of their SQL mirrored data in the Fabric environment. Get it now: SQL2Fabric-Mirroring Embedded workload integration: Striim embeds directly into the Microsoft Fabric Workload Hub, ensuring real-time data movement orchestration without leaving the Microsoft Fabric environment. Enterprises can leverage this integration for cohesive data workflows and enhanced discoverability. Real-Time data replication & streaming: Striim offers sub-second latency ingestion from SQL Server, Oracle, PostgreSQL, MongoDB, and Databricks. These structured pipelines optimize data for Azure OpenAI, Fabric Copilot, and Vector Search—delivering AI-ready insights. Enterprise security & performance: With end-to-end encryption, access control, and high-throughput event streaming via change data capture (CDC), Striim ensures secure, scalable, and low-latency performance across hybrid environments. Integrated workload with Fabric: Striim seamlessly embeds within the Microsoft Fabric Workload Hub, enabling enterprises to orchestrate and automate real-time data movement without leaving the Fabric ecosystem. AI-Optimized streaming: Striim delivers structured data pipelines tailored for Azure OpenAI, Fabric Copilot, and Vector Search. It ensures fresh, AI-ready data for machine learning models. Use cases Hybrid Cloud migration & replication: Continuously mirror on-premises SQL Server databases into Fabric OneLake for real-time analytics. Cross-Cloud data synchronization: Seamlessly integrate data across Azure, AWS, and GCP, reducing data silos. AI-Driven insights & automation: Provide fresh data pipelines for AI-powered recommendations, fraud detection, and predictive analytics. Fabric Workload automation: Utilize the Workload Hub to streamline AI-powered data transformations and event-driven automation. Closing We hope that you enjoy the update! Be sure to join the conversation in the Fabric Community and check out the Fabric documentation to get deeper into the technical details. As always, keep voting on Ideas to help us determine what to build next. We are looking forward to hearing from you!Fabric September 2025 Feature Summary
Welcome to the Fabric September 2025 Feature Summary! This month’s update is packed with exciting enhancements, such as new certification opportunities, the Power BI DataViz World Championships at FabCon Vienna, and major advancements in the Fabric Platform. Highlights include the general availability of the Govern Tab and Domains Public APIs and expanded Microsoft Purview protection and data loss prevention policies. Dive in to discover the latest improvements designed to empower your data experience. Contents Events and Announcements Get certified in Microsoft Fabric Power BI DataViz World Championships – happening live at FabCon Vienna! Fabric Platform Govern Tab in OneLake Catalog (Generally Available) Domains Public APIs (Generally Available) Microsoft Purview Protection Policies for Fabric (Generally Available) Default Sensitivity Label for Domains (Generally Available) Microsoft Purview Data Loss Prevention Policies for Fabric (Generally Available) Variable library (Generally Available) New Design for Deployment pipeline (Generally Available) New Resources and Data Sources Added to the Terraform Provider for Microsoft Fabric Fabric CLI is now Open Source! Fabric CLI v1.1.0 is here Introducing Fabric MCP (Preview) Fabric Extensibility Toolkit (Preview) Workspace-Level Workload Assignment (Preview) Fabric Multitasking Gets a Developer-Friendly Upgrade (Preview) OneLake OneLake Catalog secure tab Data Engineering Fabric User Data Functions (Generally Available) New Features in Fabric User Data Functions Notebook Integration with User Data Functions (General Available) Fabric Materialized Lake Views (MLV)– New features Python Notebook (Generally Available) NotebookUtils new APIs (Generally Available) Advanced Python intellisense – Pylance on Notebook (Generally Available) Python notebook real-time resource usage monitoring Environment Public APIs (Generally Available) Query Mirrored Databases in Spark Notebook Download Files in Lakehouse Explorer Multi-Lakehouse experience Fabric Spark Monitoring APIs (Generally Available) Fabric Spark Run Series Analysis (Generally Available) Fabric Spark Applications Comparison (Preview) Data Science Mirrored database support for Fabric Data Agent CI/CD support in Fabric Data Agent Consuming Fabric Data Agent in external applications with Python client SDK (Preview) Get feedback on example queries using the Fabric Data Agent SDK Discover which query examples influenced the Data Agent response Download Diagnostics in Data Agent Apply AI Functions in Data Wrangler (Preview) AI-Powered Code Generation and Translation Now Available in Data Wrangler (Generally Available) Data Warehouse MERGE Transact-SQL (Preview) Migration Assistant for Fabric Data Warehouse (Generally Available) Databases Copilot & Query Editor Enhancements MSSQL extension for VS Code – Fabric Integration (Preview) Point-in-Time Restore (PITR): Precision Recovery Git integration: system object references & shared queries Database Definition Import/Export via REST API Performance Dashboard Improvements Real-Time Intelligence Introducing Maps in Fabric Azure Monitor Logs Integration in Fabric via Eventstream Workspace Private Link for Eventstream’s Select Sources & Destinations Activator Just Got 10x More Powerful (Preview) Anomaly detection in Real-time Intelligence (Preview) Data Factory Dataflow Gen2 Accelerate Dataflow Gen2 Authoring with Preview-Only Steps (Preview) Modern Evaluator (NetCore-based)for Dataflow Gen2 with CI/CD (Preview) Partitioned compute for Dataflow Gen2 (Preview) Fabric Variable libraries in Dataflow Gen2 with CI/CD (Preview) Parameterized Dataflow Gen2 using public parameters mode (Generally Available) New Discover Dataflow Gen2 Parameters API (Preview) Dataflow Gen2 Support for Incremental refresh to Lakehouse as a data destination (Generally Available) New Dataflow Gen2 data destinations (Generally Available) Schema Support in Dataflow Gen2 Destinations: Lakehouse, Fabric SQL and Warehouse (Preview) Dataflow Gen2 - Natural language to custom column with Copilot (Generally Available) Dataflow Gen2 – Explain query and query steps using Copilot (Generally Available) Copilot in Modern Get Data (MGD) for Dataflow Gen2 (Preview) Dataflow Gen2 – Postgre SQL Connector adds support for Microsoft Entra ID Authentication 2-Tier Pricing Model for Dataflow Gen2 (CI/CD Pipelines Microsoft Fabric Data Factory Data pipelines are now ‘pipelines’ New Activities in Email and Teams (Generally Available) Dataflow activity in pipelines: New parameters experience Debug your pipeline Expression with the Evaluate Expression experience Functions Activity with User Data Functions (Generally Available) Add up to 20 schedules for your pipeline! On-premises and VNet data gateway support for Invoke pipeline and Semantic model refresh activities Copy job Activity (Preview) Invoke pipeline activity (Generally Available) Support for Workspace Identity Variable Library integration with pipelines (Generally Available) Azure Databricks Jobs activity (Generally Available) Apache Airflow Job New Fabric Airflow features make it easy to build DAGs Fabric Apache Airflow Job now supports CI/CD Copy job Data Replication from Fabric Lakehouse with Delta Change Data Feed (Preview) Connection Parameterization with Variables library for CI/CD (Preview) Merge data into Snowflake More Connectors, More Possibilities Simplified Copy Assistant, Powered by Copy job Connectivity New connectors available in Data Factory Salesforce & Salesforce Service Cloud connector: Partition auto detection Support upsert to delta table with Lakehouse connector (Preview) Support delta column mapping and deletion vector with Lakehouse connector Support varchar(max) in table creation with Data Warehouse connector (Preview) DB2 connector: support to specify for package collection Snowflake connector: support to specify for role Gateways Virtual Network Data Gateway supports Fabric pipeline and Copy job (Generally Available) Mirroring Mirroring for Google BigQuery (Preview) Mirroring for Oracle (Preview) Mirroring for Azure SQL Managed Instance (Generally Available) Mirroring support for sources behind a firewall (VNET and OPDG) Azure SQL Database mirroring now supports Workspace Identity authentication Developer tooling Fabric VS Code extension (Generally Available) Extensibility Lucid Data Hub’s Fabric Workload (Preview) Statsig’s Experimentation Analytics on the Fabric Workload Hub (Preview) Events and Announcements Get certified in Microsoft Fabric Join the thousands of other Fabric users who’ve achieved over 50,000 certifications collectively for the Fabric Analytics Engineers and Fabric Data Engineers roles. To celebrate FabCon Vienna, we are offering the entire Fabric community a 50% discount on exams DP-600, DP-700, DP-900, and PL-300. Request your voucher. Power BI DataViz World Championships – happening live at FabCon Vienna! Four finalists are taking the stage at FabCon to compete for the title of world champion! FabCon Vienna Power BI DataViz World Championships – and the winner is... Congratulations to Paulo Grijó! Read more about the finals and all four finalists. Fabric Platform Govern Tab in OneLake Catalog (Generally Available) In today’s data-driven world, effective data governance is crucial for ensuring the integrity, security, and usability of data. We’re excited to announce the general availability of the governance experience within the OneLake catalog. With this experience we’re empowering individual data owners with the tools and insights they need to govern and secure their data estate within Fabric. Additionally, you can now chat with your data easily with copilot: In the ‘view more’ report, select the ‘Copilot’ icon and start chatting with your data to gain more insights, drill through to get more details on areas of interest and get overall summary of trends surfaced in the report. To learn more, refer to the documentation for - OneLake catalog overview, Governance in OneLake catalog, and Governance and compliance in Fabric. Domains Public APIs (Generally Available) Microsoft Fabric data mesh architecture / federated architectures support organizing data into domains and sub domains, helping admins to manage and govern the data per business context with various delegated settings and the ability to create dedicated tags. Domains and sub domain structures enable data consumers to filter and discover content from the area most relevant to them. Previous APIs will remain available until March 31 st , 2026. To learn more about Domains - REST API (Admin), and Domains, refer to the documentation. Microsoft Purview Protection Policies for Fabric (Generally Available) Microsoft Purview protection policies provide a powerful and automated way to enforce data governance and security within Microsoft Fabric. These policies leverage sensitivity labels from Microsoft Purview Information Protection to automatically restrict access to sensitive data assets, like Lakehouses or KQL databases, within Fabric. Instead of manually configuring access controls for every data item, organizations can define a single policy that applies a specific level of protection to all data with a given sensitivity label. By integrating directly with Fabric, these policies allow you to apply granular controls, like blocking access for all but a select group of users, ensuring that only authorized individuals can interact with your most critical data. This automated and centralized approach reduces the burden on data engineers and IT admins, enabling them to focus on building data solutions while the platform handles the enforcement of security and compliance standards. Refer to the Protection policies in Microsoft Fabric (preview) documentation to learn more. Default Sensitivity Label for Domains (Generally Available) Once defined, these labels apply automatically to all new items created in the specified domain, allowing you to use a centralized configuration to reduce human error and ensure consistent data handling across the organization. Refer to the Domain-level default sensitivity labels documentation to learn more. Microsoft Purview Data Loss Prevention Policies for Fabric (Generally Available) In Microsoft Fabric, Data Loss Prevention (DLP) policies are a game-changer for organizations by acting as a proactive defense for sensitive data. These policies automatically identify and protect confidential information—such as personally identifiable information (PII), financial data, or intellectual property—as it's created or moved within the Fabric environment, including semantic models and structured data in OneLake, such as Lakehouses, mirrored databases and more. This capability is critical for ensuring compliance with various regulations like GDPR and HIPAA, and it helps to prevent costly data breaches that can lead to significant financial loss and reputational damage. By providing real-time alerts to administrators and policy tips to users, DLP policies empower a culture of security awareness and reduce the risk of both accidental and malicious data leaks. With this release we are announcing the general availability of DLP policies for your OneLake data, which means businesses can embed data security directly into their data workflows, moving beyond reactive measures to a more secure, proactive posture. Refer to the documentation, to learn how to Get started with data loss prevention policies for Fabric and Power BI. Variable library (Generally Available) Beginning September 30th, the Variable library item will officially be Generally Available and will be supported as such in Pipelines. Additionally, support for Variable library is expanding beyond Shortcut for Lakehouse. Variable library can now also be used in: Dataflow Gen 2 - Fabric Variable libraries in Dataflow Gen2 with CI/CD (Preview) Copy job - Connection Parameterization with Variables library for CI/CD (Preview) This capability is now extended to include Copy job scenarios, allowing you to replace static source and destination values with references to Variable library variables, and in addition, Dataflow Gen2 now supports Variable library in the Query Editor, enabling parameterization of elements such as source paths and DAX expressions. Launched in April 2025, the Variable library has delivered significant value by enabling custom values and configurations across various release stages. It supports dynamic fields in data pipelines, variable in notebook code with NotebookUtils and parameterization of its default Lakehouse via %%Configure, and Lakehouse Shortcut sources. Ready to streamline your pipeline and notebook configurations? Start using Variable library and explore how it can simplify your workflows. New Design for Deployment pipeline (Generally Available) The new design for Deployment pipelines will be generally available beginning September 30th, 2025. As part of this rollout, the previous design will be deprecated and is scheduled for removal in the upcoming quarter. A formal notice will be issued 30 days in advance to ensure a smooth transition. New Resources and Data Sources Added to the Terraform Provider for Microsoft Fabric Introducing additional resources and data sources in the Terraform Provider for Microsoft Fabric, enabling broader Infrastructure as Code coverage. With these additions, teams can automate even more aspects of their Fabric environment using Infrastructure as Code (IaC). Newly Added Resources and Data sources Connections OneLake Shortcut implementation Dataflow Gen2 implementation Digital twin builder implementation Apache Airflow Job implementation Deployment Pipeline implementation Copy job implementation Mounted Data Factory implementation Folders implementation Deployment Pipeline Role Assignment Warehouse snapshot By expanding the set of supported resources, the Terraform Provider for Fabric makes it easier to: Standardize automated deployment for Fabric. Strengthen governance and security with automated role assignments. Enable collaboration by treating Fabric configurations as versioned code. Getting Started Upgrade to the latest version of the Terraform Provider for Microsoft Fabric. 2. Check the provider documentation for syntax and usage examples. 3. Add these new resources to your Terraform configuration (main.tf) to start managing them as code. Fabric CLI is now Open Source! Fabric CLI provides developers with a fast, scriptable, and intuitive way to navigate and operate Microsoft Fabric — whether locally or in CI/CD pipelines. Since its debut earlier this year, it’s already reshaping how teams automate workflows and manage their data estate. Now, we’re taking the next step by opening it up to the community. This isn’t just about sharing code — it’s about unlocking the innovation of our developer community. The CLI is built with AI-assisted development in mind, so contributors can move faster than ever: surfacing real needs, building new capabilities, and shaping the CLI around what matters most to them. We believe the best developer tools are built with the community, not just for it. So, if you’ve got an idea, a use case, or a feature request, jump in. Open an issue, suggest a feature, or contribute directly. Example of creating an issue in the newly open-sourced repo of Fabric CLI Check out the Fabric CLI repo. Let’s build the future of Fabric CLI, together. Fabric CLI v1.1.0 is here We’ve shipped v1.1.0 with a focus on usability, reliability, connectivity, and groundwork for AI-assisted contributions based on your top asks. Output formatting, JSON - machine-readable output for automation and pipelines. Folder support, organize items in folders/subfolders with predictable, path-like ops. Command & argument autocomplete faster, fewer typos in interactive use. Context persistence (command-line mode) cd once; subsequent commands run in context. Support workspace private links tighter network boundaries for secure environments. AI-assisted contributions, foundations that make it easier for AI agents (and humans!) to add new commands and capabilities. Plus, a round of quality fixes and safety improvements. Banner of v.1.1.0 of the Fabric CLI, introducing new features and capacities. Refer to the Full changelog. Introducing Fabric MCP (Preview) Fabric MCP, is a developer-focused Model Context Protocol server that enables AI-assisted code generation and item authoring in Microsoft Fabric. It streamlines how developers build around Fabric’s public APIs and create Fabric items using built-in templates and best-practice instructions — reducing coding time, minimizing errors, and boosting productivity. Designed for agent-powered development and automation, it integrates with tools like VS Code and GitHub Codespaces as part of the Microsoft MCP initiative — and is fully open and extensible. Asking GitHub Copilot in VS Code to generate a Python script using the Fabric MCP. Get started with the: Catalog of official Microsoft MCP (Model Context Protocol) server implementations for AI-powered data access and tool integration. Fabric Extensibility Toolkit (Preview) The Microsoft Fabric Extensibility Toolkit is the next evolution of the Workload Development Kit. The new toolkit represents a significant step forward in enabling organizations to create and integrate data applications that show up in Microsoft Fabric. Now organizations and software development companies can build Fabric items within days or hours if they use the Copilot optimized Starter Kit. By simplifying the development process and providing robust integration capabilities with the platform, the Extensibility Toolkit enables you to bring your data applications to Fabric Workspaces where your data and users are. The Extensibility Toolkit builds on the foundation of the Workload Development Kit while introducing several key improvements and new capabilities. Key advantages leveraging the toolkit: Bring Your App to Fabric: Publish your organization’s data applications directly into Fabric workspaces as custom items Leverage Your Platform: Seamlessly integrate Power BI reports, Spark jobs, the Fabric SaaS foundation and more Rapid Development low cost to maintain: Launch in hours with LLM-ready samples and streamlined tooling We look forward to seeing what you'll build with the Microsoft Fabric Extensibility Toolkit. Whether you're creating specialized data applications, custom visualizations, or data integrations, the Extensibility Toolkit will make your journey straightforward. As part of this launch, we have also created a new Fabric Community Repository. This repository will contain a wide variety of item types built with the Extensibility toolkit you can add to your tenant. The first release contains: Package Installer allows users to deploy configured items (e.g. Lakehouse, Notebook, etc.) with definitions, data, shortcuts and much more. OneLake Explorer can be used to view and edit the OneLake Storage of any Fabric data item on the platform directly. Workspace-Level Workload Assignment (Preview) This capability allows workspace admins to add additional workloads directly to their workspaces, eliminating the need for tenant or capacity-level setup. Key highlights include: Assign workloads from the Workloads Hub directly to a workspace. Enable multiple workloads in a single workspace without impacting the rest of the tenant. Maintain governance and security with tenant-level controls and Entra ID consent. This update gives teams more flexibility to innovate while preserving organizational control. Refer to the documentation for how to Add a workload in the workload hub. Fabric Multitasking Gets a Developer-Friendly Upgrade (Preview) We’ve rolled out a set of UI enhancements in Fabric aimed at making multitasking smoother and more intuitive. Inspired by modern IDEs, these updates include: Horizontal tabs for open items with clear labels. Support for multiple active workspaces with color coding and numeric labels. A new Object Explorer for structured navigation across open workspaces. The open item limit has been increased beyond the previous cap of 10, providing enhanced flexibility for developers managing multiple resources simultaneously. These changes address common pain points around navigation, context switching, and multitasking — making Fabric more aligned with how developers work every day. These improvements apply only to the Fabric experience, and do not affect the Power BI experience. Learn more about the new developer friendly experience. OneLake OneLake Catalog secure tab The OneLake catalog now features a dedicated Secure tab, offering a centralized view of security settings across your Fabric items. The Secure tab provides two powerful lenses into your security setup: users and security roles. The View Users page combines user permissions across workspaces, letting you identify users with privileges they shouldn’t have. The View Security Roles page gives a holistic view of OneLake security roles across workspaces and item types. You can easily view existing OneLake security roles and even make updates to them inline. Powerful filtering options make it easy to find the exact roles and items you are looking for. It’s a streamlined experience designed for transparency and control. You can get started with the Secure tab today or check out OneLake security access control model (preview) documentation. Data Engineering Fabric User Data Functions (Generally Available) This feature provides a Fabric-native platform to host and run your business logic using Python functions. With Fabric User Data Functions, you can do the following: Easily create and publish functions to run on Fabric using the in-browser portal editor or with the native VS Code extension using templates and libraries. Connect to your Fabric data sources and create data applications. Enhance your Fabric Notebooks and Data pipelines with custom logic. Create Translytical Taskflows using Power BI. New Features in Fabric User Data Functions With Fabric User Data Functions now generally available, the following are the new features we’re introducing: Test your functions using Develop mode: This feature allows you to execute your functions in real-time before publishing them. OpenAPI spec generation in Functions portal: You can access the OpenAPI specification for your functions using the Generate code feature in the Functions portal. Async functions and pandas support: You can now create async functions to optimize the execution for multi-task functions. Additionally, you can now pass pandas DataFrame and Series types as parameters to your functions using the Apache Arrow format. To learn more, refer to the What is Fabric User data functions (Preview)? documentation. Notebook Integration with User Data Functions (General Available) Building on UDF in preview, we've made considerable advancements to native support for pandas DataFrames and Series, powered by Apache Arrow. With the built-in UDF capabilities in NotebookUtils, you can: Browse all functions within a UDF item. Invoke specific functions directly from your notebook. Explore metadata such as parameters, signatures, and return types with helper methods. Quickly discover functions using IntelliSense and autocomplete. Work seamlessly across multiple languages, including Python, PySpark, Scala, and R. Additionally, we are introducing native support for pandas DataFrames and Series, enabled by deep integration with Apache Arrow. Faster performance with Arrow-optimized data handling. Scalability to process large-scale datasets. Seamless compatibility with existing pandas workflows. You can now pass pandas DataFrames directly into a UDF, operate on them efficiently, and return results - all with minimal overhead. To learn more, refer to the Use Fabric User Data Functions with pandas DataFrames and Series in Notebooks documentation. Fabric Materialized Lake Views (MLV)– New features Materialized Lake Views (MLV) in Microsoft Fabric have received several powerful enhancements designed to improve refresh performance, lineage visibility, environment customization, and operational flexibility. These updates help organizations optimize data workflows, ensure transparency, and tailor resource usage to specific workload needs. Optimal Refresh: Enhance refresh performance by automatically determining the most effective refresh strategy—incremental, full, or no refresh—for your Materialized Lake Views. Lineage Enhancements: View lineage with source entities across Lakehouses in a Workspace, making it easier to trace data origins and dependencies. Custom Environments for MLV Refresh: Associate specific environments and tailor configurations for varied workload needs, allowing you to optimize performance and resource usage during MLV refreshes. Run on Demand: Perform refreshes on demand for the lineage without scheduling, available via both APIs and the Manage MLV UI. Ready to get started, refer to the Materialized Lake Views (MLV) documentation. Python Notebook (Generally Available) Python Notebook is a pure Python experience built on top of Fabric notebook, designed for data analysis, visualization, and machine learning, providing a smooth Python coding and execution environment. Key features Multiple built-in Python kernels: Built-in Python 3.10 and 3.11 with native features like iPyWidget and magic commands. Users can easily switch kernels to match project needs. Cost effective: Runs on a single-node cluster (2 vCores / 16 GB) for small-scale data exploration. Lakehouse & Resources are natively available: Native integration with Fabric Lakehouse and built-in resources, with drag-and-drop code generation. Mix programming with T-SQL: Python notebook offers an easy way to interact with Data Warehouse, SQL endpoints and SQL database. Users can use cell magic %%tsql or line magic %tsql to run T-SQL queries on Python runtime. Support for popular data analytics libraries: Include DuckDB, Polars, Scikit-learn, providing a comprehensive toolkit for data manipulation, analysis, and machine learning. Advanced intellisense: Powered by Pylance and Fabric’s language services for a modern coding experience. NotebookUtils & Semantic link: APIs for leveraging Fabric and Power BI in code-first workflows. Rich Visualization Capabilities: Built-in table/chart preview plus Matplotlib, Seaborn, Plotly, and PowerBIClient. Common Capabilities for Fabric Notebook: All the Notebook level features are naturally applicable for Python notebook, such as editing features, AutoSave, collaboration, sharing and permission management, Git integration, import/export, etc. Full stack Data Science Capabilities: Supports Data Wrangler, MLflow, and Copilot for advanced analytics and ML. For more details, please refer to the documentation for Use Python experience on Notebook. NotebookUtils new APIs (Generally Available) These APIs have been designed to streamline and enhance your workflow by providing powerful programmatic capabilities for managing notebooks and lakehouse assets, as well as improving performance and usability. RunMultiple: You can use notebookutils.notebook.runMultiple() to run multiple notebooks in parallel or in a predefined topological order. It uses a multi-threaded execution model within the same Spark session, all notebook runs share the same cluster, which can significantly improve compute resources utilization. Fastcp: The notebookutils.fs.fastcp() provides a more efficient alternative to the traditional cp command. For heavy data workloads in Fabric, using fastcp is highly recommended to boost performance and save time. Notebookutils.notebook CRUD APIs: You can manage Notebook items programmatically—create, read, update, and delete them—without manual steps. These APIs make it simple to integrate notebook operations directly into your data workflows. Lakehouse utilities: notebookutils.lakehouse provides CRUD capabilities for Lakehouse assets such as tables and files. These utilities provide create, read, update, and delete operations, allowing you to focus on building solutions. Runtime.context: With notebookutils.runtime.context you can get the context information of the current live session, including the notebook name, default lakehouse, workspace info, if it's a pipeline run, etc. For more details, please refer to the documentation for NotebookUtils (former MSSparkUtils) for Fabric. Advanced Python intellisense – Pylance on Notebook (Generally Available) This integration enhances Python development by providing intelligent code completions, precise error detection, and comprehensive code insights. It is supported for both PySpark environment and Python environment of notebook, and by default enabled for all notebook users. Key highlights include: Smarter and more relevant autocompletion Enhanced support for lambda expressions Rich parameter suggestions Detailed hover information Improved docstring rendering Precise error highlighting Pylance enhances the efficiency, accuracy, and overall experience of writing Python and PySpark code on Fabric notebooks, enjoy coding with Fabric! Python notebook real-time resource usage monitoring With the resource monitor pane, you can now track critical runtime information such as session duration, compute type, and real-time resource metrics—including CPU and memory consumption—directly within your notebook. This feature offers a clear and immediate overview of your active session, helping you stay informed about the resources your code is consuming as you work. This enhancement provides better visibility into how your Python workloads are utilizing system resources, making it easier to optimize performance, control costs, and avoid unexpected out-of-memory (OOM) errors. By monitoring these metrics in real time, you can quickly identify resource-intensive operations, understand usage patterns, and make informed decisions about scaling or modifying your code. To start using it, simply ensure your notebook language is set to Python and start a session. The resource usage monitor will appear as a pane within the notebook interface, providing a seamless and integrated monitoring experience for all users working with Python code in Fabric notebooks. Environment Public APIs (Generally Available) This brings a new set of capabilities, improved contracts, and a migration and deprecation plan for existing APIs. What’s new? Create, get or update environment with definition. Import, export or remove external libraries (staging & published state). Upload or delete custom library. How to migrate? Existing APIs with contract update Some APIs have updated response contracts (e.g., Publish environment, List staging/published libraries/Spark settings). A new query parameter ‘preview’ is introduced to facilitate the transition of request/response contract changes. The ‘preview’ query parameter defaults to ‘True’ until March 31, 2026, making the preview contracts still available. For migration, add the query parameter preview=False to start using the new GA contract. Deprecation Two preview APIs (Upload staging libraries and Delete staging libraries) will be deprecated on March 31, 2026. Please migrate to the new APIs as soon as possible. For a full list of impacted APIs and migration guidance, please refer to the documentation for Manage the environment through public APIs. Query Mirrored Databases in Spark Notebook You can now connect and query mirrored databases (MirrorDBs) directly from your Spark notebooks, making it easier than ever to analyze data across your enterprise. This update allows you to: Add MirrorDBs as data sources in your notebook and run read-only Spark queries on mirrored database tables using SparkSQL or PySpark—just like you do with lakehouse data. The supported MirrorDBs including Azure Cosmos DB, Azure SQL Database, Snowflake and Open Mirroring. Coming soon, the support for other MirrorDBs. No need to attach a default Lakehouse to run read-only queries; simply use fully qualified four-part names (workspace.database.schema.table) for precise querying and compatibility. Join MirrorDB tables with lakehouse tables for deeper insights across all your Fabric data. This feature expands your analytics capabilities and lets you securely access and analyze mirrored data alongside lakehouse assets, all in one place. Download Files in Lakehouse Explorer Download files directly from any Lakehouse item, empowering you to work more efficiently, reduce friction in your data workflows, and gain faster insights. Capabilities Download files from both table files and the File section (with required permission). Keep your data secure and compliant by including Microsoft Information Protection (MIP) sensitivity labels for supported files. OneLake data export must be enabled for your tenant to use this feature. Multi-Lakehouse experience It's now easier than ever to organize and access your data. You can now add, view and manage multiple lakehouses in a single unified view. Capabilities Quickly add multiple reference lakehouses that you have permission to ensuring secure collaboration. Work with several lakehouses in a consolidated view, with clear distinction between your primary and reference lakehouses. Sort, filter, and search across all connected lakehouses, schemas, tables, and files for faster data discovery. Perform key actions like previewing data, creating subfolders, renaming, and deleting objects—all from one place. This feature boosts productivity and gives you the flexibility to manage complex data estates, all without switching between different views or tools. Fabric Spark Monitoring APIs (Generally Available) Introducing advanced observability features and improved automation for managing Spark workloads in Microsoft Fabric! New APIs for Single Spark Applications. Spark Advisor API - Provides recommendations and skew diagnostics to help identify bottlenecks and optimize performance. Resource Usage API - Offers granular metrics on vCore allocation and utilization for executors within a Spark application. Advanced Filtering Support - The workspace-level API now supports filtering capabilities to help users narrow down applications by time range, submitter, application state (Succeeded, Failed, Running), and more! This enhancement allows for more efficient analysis and targeted troubleshooting in large-scale environments. New Application-Level Properties for Deeper Insight - To support more transparent resource planning and monitoring, the following properties have been added to Spark Monitoring APIs: Driver Cores & Memory Executor Cores & Memory Number of Executors Dynamic Allocation Enabled Dynamic Allocation Max Executors These new fields help teams better understand and optimize their Spark resource allocations. The Fabric Spark Monitoring APIs are now production-ready—providing a comprehensive solution for monitoring, diagnosing, and optimizing Spark workloads in a scalable, automated fashion. Fabric Spark Run Series Analysis (Generally Available) We’ve introduced a series of enhancements based on customer feedback and product readiness. These include improved accessibility through UI refinements that meet compliance standards, support for analyzing Spark applications while they’re still running, and a major upgrade to the anomaly detection infrastructure for more accurate and scalable outlier identification. Fabric Spark Run Series Analysis is an advanced tool for understanding, comparing, and optimizing recurring Spark job executions. It now offers enhanced capabilities, greater accessibility, and a robust foundation to support enterprise-scale performance tuning. Key Capabilities Run Series Comparison - Compare the execution duration of a Spark run against historical runs within the same series. Drill into input/output data differences to identify root causes of performance variation. Outlier Detection and Analysis - Automatically detect anomalous runs within a series and surface potential contributing factors—such as resource constraints or configuration changes. Detailed Run Instance View - Explore individual run instances to access detailed time distribution metrics, offering insights into each phase of execution. Configuration values—both user-defined and auto-tuned—are also surfaced for reference and optimization. Fabric Spark Applications Comparison (Preview) This new capability enables developers and data engineers to analyze, debug, and optimize Spark performance across multiple application runs—whether you're evaluating the impact of code changes or data variations. The Spark Applications Comparison feature lets users select and compare up to four Spark application runs side by side. By visualizing and contrasting key execution metrics, you can quickly pinpoint performance regressions, improvements, and anomalies. Key Benefits Compare runs from the same artifact (Notebook or Spark Job Definition) Detect performance regressions or gains by analyzing metric deltas against a baseline run. Troubleshoot issues using detailed insights into execution time, I/O data trends, and resource utilization. Data Science Mirrored database support for Fabric Data Agent Microsoft enables users to directly connect mirrored database artifacts including Azure Cosmos DBs, Azure SQL, Oracle, Snowflake, Databricks, and other databases using open mirroring. Including This integration allows users to leverage Data Agent’s Natural Language to SQL (NL2SQL) capability, so users can ask questions in plain English, and receive LLM- powered insights across their data estate. Key benefits include broad compatibility, direct integration, customizable and scoped knowledge, and seamless bridging between external data and AI applications within Fabric. The feature streamlines analytics and empowers smarter, faster decision-making by unlocking AI-driven insights. CI/CD support in Fabric Data Agent Fabric Data Agents now support CI/CD, ALM flow, and Git integration, enhancing management, version control, and collaboration for Data Agent artifacts. These features promote reliable, scalable, and auditable development practices by enabling systematic management of changes, dedicated workspaces for development stages, and broad data source support. Git integration tracks all modifications, supports branching for independent experimentation, and enables controlled merging, improving teamwork and allowing quick reversion if issues arise. To get started with CI/CD, ALM flow, and Git integration for your Fabric Data Agent, refer to the Fabric Data Agent documentation for step-by-step instructions. Consuming Fabric Data Agent in external applications with Python client SDK (Preview) This new feature empowers developers to integrate Fabric Data Agents into custom web apps and workflows, enabling natural language querying, automated reporting, and embedded insights—all while respecting user identity and permissions via Microsoft Entra ID. Setup in Visual Studio Code, including cloning the sample client, configure authentication with InteractiveBrowserCredential, and use the DataAgentClient to interact with the agent. It’s a streamlined, developer-friendly way to extend Fabric’s intelligence beyond its native environment. Explore this guide for step-by-step instructions. Get feedback on example queries using the Fabric Data Agent SDK The Fabric Data Agent SDK now offers evaluate_few_shot_examples(), a tool for creators to get structured feedback on their example queries. For each example, the function assesses clarity (determining if the natural language question is clear and unambiguous), mapping (checking if the SQL query accurately reflects the intent of the question), and relatedness (verifying that all literals in the question are correctly mapped to corresponding literals in the SQL). The evaluation then provides a final reasoning summary on the overall quality of each example query. With this capability, creators can refine their examples more effectively, ensuring their data agents deliver accurate, intuitive, and high-quality responses. Want to learn more? Check out the Evaluate a Fabric Data Agent documentation. Discover which query examples influenced the Data Agent response When the Data Agent answers a question, it reviews the example queries you’ve provided and uses them to guide its reasoning. With our latest update, creators can view exactly which example queries were used during a run-step to help shape the agent’s response. Behind the scenes, the agent searches all configured examples and selects the top 3 most relevant ones based on the user’s question. These examples are passed to the model as reference points when generating a query. Now, within the chat canvas, you’ll be able to view those selected examples, giving you more transparency into how the agent is grounding its response and making it easier to improve your configurations over time. Download Diagnostics in Data Agent Download a diagnostics file for any run step in the Data Agent chat canvas—giving you clear visibility into how the agent processed your question behind the scenes. The file includes details like which tools were used, how the question was interpreted, the intermediate reasoning steps, and any errors or fallback logic that occurred. A powerful way to troubleshoot issues or provide rich context when working with Microsoft support. To use it, simply select the ‘Download Diagnostics’ button in the chat canvas. The file is fully viewable and editable, so you can review and clean up the contents before sharing as needed. For more information, refer to the Evaluate your data agent (preview) documentation. Apply AI Functions in Data Wrangler (Preview) The ability to apply AI functions directly within Data Wrangler to transform your data quickly and visually is now available. AI functions allow you to perform tasks like text summarization, classification, translation, sentiment analysis, grammar correction, or your own prompt, all without writing complex code. See your transformations instantly with real-time previews, so you can make quick adjustments if needed. A GIF showing how to use AI functions in Data Wrangler to instantly classify transactions on a bank statement data frame. Simply select the desired function from AI Enrichments within the Data Wrangler interface. The results are interactive and editable, giving you full control over your data transformations while saving time and reducing manual effort. AI-Powered Code Generation and Translation Now Available in Data Wrangler (Generally Available) Accelerate data prep with AI-powered capabilities in Data Wrangler. In the visual interface, you can see smart suggestions from Microsoft PROSE for operations that are relevant to your data frame. Describe your desired transformations in natural language, and Copilot will generate code and an instant preview of the results. A GIF showing how to apply AI-based suggestions, use Copilot to generate code from a natural language description, and automatically translate pandas code to PySpark. For big data workflows, Data Wrangler can translate all your pandas operations back to PySpark, making it easy to scale your transformations across large datasets. Data Warehouse MERGE Transact-SQL (Preview) This command blends INSERT, UPDATE, and DELETE operations all into a single statement based on your specified conditions between two tables, improving readability and providing a uniform standard for transformations across your ETL jobs. For more details, refer to the MERGE (Transact-SQL) documentation, and try using MERGE today! Migration Assistant for Fabric Data Warehouse (Generally Available) This AI powered assistance helps you migrate your analytical warehouse or database from Azure Synapse Analytics dedicated SQL pool or SQL server database used for analytics work to Fabric Data Warehouse using a DACPAC file. For more details, refer to the Migration Assistant for Fabric Data Warehouse documentation. Databases Copilot & Query Editor Enhancements Introducing several enhancements to Copilot in SQL in Fabric and Query Editor: Connect in SSMS with One Click: Instantly connect to your Fabric SQL database in SQL Server Management Studio (SSMS) without manual entry—streamlining developer workflows and reducing friction for data professionals. Bulk Query Management: Delete multiple queries at once by simply holding Shift, selecting, and right-clicking, making workspace cleanup fast and intuitive. Share Queries Across Teams: Create shared queries that any admin, contributor, or member in your workspace can view and edit, supporting collaborative development and knowledge sharing. Copilot Mode Selector: Easily toggle between read-only and read/write modes. In read-only, Copilot generates SQL code but doesn’t execute changes; in read/write, Copilot executes code only after your approval, ensuring safe automation and governance. To learn more about these features, refer to the Microsoft Copilot in Microsoft Fabric in the SQL Database Workload Overview documentation. MSSQL extension for VS Code – Fabric Integration (Preview) The latest version of the MSSQL extension for VS Code introduces Fabric connectivity and provisioning (Public Preview), bringing SQL database in Fabric directly into your editor. Instead of switching to the Fabric Portal, copying connection strings, or creating databases outside VS Code, you can now authenticate with Microsoft Entra ID, browse workspaces in a tree view, and connect instantly. You can also create a new SQL database in Fabric in just a few steps and start querying in under three minutes—eliminating context switching and making it faster to prototype and build modern apps. Key highlights Fabric Connectivity: Sign in with Microsoft Entra ID and connect to Fabric workspaces directly from the Connection dialog. Workspace Search & Browse: Navigate workspaces and resources in a tree view with built-in search for faster discovery. Fabric Provisioning: Provision a SQL database in Fabric from the Deployments page and connect instantly in VS Code. Cross-extension Flow: Launch connections from the Fabric extension or Portal using the 'Open in MSSQL' option. Frictionless Development: Reduce context switching with a fully in-editor workflow for connecting, provisioning, and querying Fabric databases. Figure: Fabric browser experience in the MSSQL extension for VS Code Watch the SQL database in Fabric connectivity and provisioning demos in MSSQL for VS Code and check MSSQL Extension for VS Code: Schema Compare, Schema Designer, Local SQL Server Container GA documentation to learn more. Point-in-Time Restore (PITR): Precision Recovery PITR allows users to restore a database to any specific moment within the configured retention window. Whether recovering from a user error, application bug, or security incident, PITR provides granular control to rewind your database to a known good state. Until recently, PITR in Fabric SQL DB supported a 7-day retention window. With the latest update, this can now be extended to 35 days, offering significantly more flexibility for operational recovery and compliance scenarios. To learn more about this feature, refer to the Restore a database from a backup documentation. Git integration: system object references & shared queries Fabric SQL’s Git integration is now rolling out, including the following features: Validation of system object references (e.g., tables/views in the [sys] schema) during local development. Tracking of shared queries, allowing teams to monitor changes over time and maintain version control across collaborative environments. To learn more about this feature, refer to the Get started with Git integration documentation. Database Definition Import/Export via REST API Fabric SQL empowers developers with REST APIs to: Export database object definitions as portable DACPAC files. Import compiled definitions (DACPAC) to update existing databases, with automatic diff detection and application of changes. To learn more about this feature, refer to the Create a SQL database with the REST API documentation. Performance Dashboard Improvements The performance dashboard now provides memory consumption metrics. This new feature offers real-time insights into memory usage by individual database queries, enabling better resource management and optimization. The dashboard now includes memory consumption alongside CPU usage, user connections, requests per second, blocked queries, database size, automatic index info, and query performance metrics for comprehensive monitoring. To learn more about the performance dashboard, refer to the Performance Dashboard for SQL database documentation. Real-Time Intelligence Introducing Maps in Fabric Geospatial insights into the Real-Time Intelligence workload are available with Maps in Microsoft Fabric. Users can ingest location data from a Lakehouse or Eventhouse, visualize it instantly, and build map-centric applications without specialized knowledge or writing code. Whether you are tracking mobile assets, analyzing campaign performance, or monitoring infrastructure, Maps adds context, reveals patterns, and helps you tell the story behind the numbers within minutes. Refer to the Start with Fabric Maps documentation to learn more, connect your data, choose your layers, and start exploring. Spatial doesn’t have to be special! Azure Monitor Logs Integration in Fabric via Eventstream Azure Monitor Diagnostic Logs now integrate directly with Microsoft Fabric via Eventstream, enabling real-time ingestion of metrics and logs from Azure resources into Fabric-native analytics workflows for immediate transformation and analysis—eliminating the need for manual data pipelines. Once ingested into Eventstream, data can be cleaned, shaped, and enriched with native tools—all without writing a single line of code—streamlining the path from raw telemetry to actionable insight. This integration provides unified observability across Azure resources and business data, empowering teams to respond quickly and make informed decisions. Logs can be routed to Data Activator for instant alerts or to Eventhouse for deeper analysis, supporting both immediate operational needs and long-term strategic goals. You can find the connector card ‘Azure Diagnostics’ under the Data Sources page in the Real-Time Hub. Select it to browse all available Azure resources and click through the configuration process to the metrics and logs you want to bring into Eventstream. Workspace Private Link for Eventstream’s Select Sources & Destinations Introducing the integration of Workspace Private Link with Eventstream, enabling secure, private connectivity between your data sources and Microsoft Fabric—without exposure to the public internet. A workspace-level private link maps a workspace to a specific virtual network using the Azure Private Link service. With this integration in Eventstream, it allows you to restrict public internet access and enforce access only through approved virtual networks via managed private endpoints. This ensures that data streaming into Eventstream is tightly controlled and protected from unauthorized access. The diagram demonstrates a typical Eventstream setup operating under Workspace Private Link. SQL DB1 connects to Eventstream and streams CDC events securely via Workspace Private Link. SQL DB2 is blocked from connecting because public access to the workspace is disabled. For more details about Workspace Private Link, refer to the Overview of workspace-level private links documentation. Activator Just Got 10x More Powerful (Preview) We’re excited to announce a major performance upgrade to Activator, our no-code, low-latency event detection engine that helps power Real-Time Intelligence in Microsoft Fabric. Activator now supports up to 10,000 events per second (EPS), a tenfold increase from the previous 1,000 EPS limit. This change unlocks new possibilities for customers working with high-frequency data. Whether you're monitoring sensor data, tracking business operations, or responding to user behavior in real time, Activator can now keep pace with even the most demanding workloads. By scaling Activator’s throughput, we’re reinforcing our commitment to delivering fast, flexible, and frictionless automation for modern data applications. If you’re already using Activator, there’s nothing you need to do. Just enjoy the 10x scale. And if you’re not yet using it, now’s a great time to start. We welcome your feedback as we continue to evolve Activator to meet the needs of real-time data applications at scale. Ready to learn more? Refer to the What is Fabric Activator? Transform data streams into automated actions documentation. Anomaly detection in Real-time Intelligence (Preview) In today’s fast-paced digital landscape, spotting anomalies in your data as they happen can make all the difference. Whether you’re a seasoned data professional or a business user keeping an eye on operations, the new anomaly detection feature in Real-Time Intelligence puts powerful insights right at your fingertips—no coding required. To get started select your Eventhouse and choose the ID fields and values to monitor for anomalies. The system analyzes your selection using a library of industry-standard models, automatically testing and recommending the best fit for your dataset. Meaning you can easily spot outliers, trends, or unexpected behavior without needing statistical expertise. Once anomaly detection is set up, you get a preview of detected anomalies right in the interface. Try out different models to see which one best highlights the patterns you’re interested in, allowing you to validate the findings before sharing or acting on them. When you’re happy with the results, publish anomaly events to the Real-Time Hub and set up alerts so you’ll be notified instantly—via Teams messages or emails—whenever an anomaly appears. With these new anomaly detection capabilities, everyone from analysts to business users can act quickly on real-time insights. The no-code interface, automatic model selection, and flexible alerts make tracking changes and unexpected events easier than ever. Data Factory Dataflow Gen2 Accelerate Dataflow Gen2 Authoring with Preview-Only Steps (Preview) Dataflow Gen2 empowers users with a visual, intuitive experience for shaping and transforming data directly within Microsoft Fabric. At the heart of this experience is the data preview pane, which evaluates each transformation step independently to ensure accurate previews tailored to your scenario. However, rendering the full schema and data values can sometimes take several seconds — or even minutes — depending on factors like data source latency, query complexity, and evaluation overhead. In many authoring scenarios, you may not need to preview the entire dataset to validate your logic. To streamline this experience, we’re introducing preview-only steps — a new capability that lets you designate specific transformation steps to run exclusively during authoring. These steps are excluded from refresh operations, allowing you to work with sample data without impacting production performance. You can enable this feature by simply clicking any step in your query and selecting Enable only in preview. This is especially useful when paired with filtering steps, enabling you to isolate a subset of your data for faster iteration and validation. Whether you're building complex transformations or exploring new data sources, preview-only steps help you stay focused, responsive, and efficient—without compromising refresh integrity. As an accelerator, when connecting to a source that provides a file system view (like Azure Data Lake Gen2, SharePoint files, Local Folder and others) you will now see a small gear icon in the top right corner that will allow you to quickly add new preview only steps. If you select the combine files option from this dialog, you will also be able to see another gear icon this time to add a preview only step to filter the sample transformation file. This new feature will help you accelerate your work during authoring time without sacrificing any runtime evaluations. We hope that you give this new feature a try and share your feedback in our Data Factory community forum. To learn more about this feature, refer to the Preview only step in Dataflow Gen2 documentation. Modern Evaluator (NetCore-based) for Dataflow Gen2 with CI/CD (Preview) The Modern Query Evaluation Engine for Dataflow Gen2 (CI/CD) is a powerful enhancement that can substantially improve the performance of query evaluation in your dataflows. This new engine is designed to deliver faster and more efficient execution, helping you scale your data transformation workflows with greater speed and reliability. The Modern Evaluator can be enabled directly from the Options dialog (Scale tab) in both existing and new Dataflow Gen2 (CI/CD) items. While support is currently limited to a subset of connectors, it already includes many commonly used sources such as Azure Blob Storage, Azure Data Lake Storage, Fabric Lakehouse, Fabric Warehouse, OData, Power Platform Dataflows, SharePoint, and Web. To learn more about this feature, refer to the Modern Evaluator for Dataflow Gen2 with CI/CD (Preview). Partitioned compute for Dataflow Gen2 (Preview) Microsoft Fabric now offers Partitioned Compute in Dataflow Gen2, enabling parallel execution of dataflow logic to significantly reduce evaluation time. This feature is perfect for handling large file sets, like those in Azure Data Lake Storage Gen2, where operations can be done simultaneously. What it Delivers Parallel Processing: Automatically partitions data sources and evaluates each partition concurrently. Supported Connectors: Azure Data Lake Storage Gen2, Fabric Lakehouse, Folder, and Azure Blob Storage. Combine Files Experience: Automatically generates partition keys to optimize performance. Using internal benchmarks, we’ve seen how it has reduced run times of some Dataflows. An example using the New York City green taxi data from 2023 (12 parquet files), it reduced the processing time from 1.5 hours to under 25 minutes when loading to a Fabric Warehouse. While results may vary, we welcome you to give this new feature a try and share your feedback with us. To learn more about this feature, refer to the Use partitioned compute in Dataflow Gen2 (Preview) documentation. Fabric Variable libraries in Dataflow Gen2 with CI/CD (Preview) Microsoft Fabric now supports Variable libraries in Dataflow Gen2 with CI/CD. This new capability introduces dynamic configuration management across environments, helping teams streamline CI/CD workflows and improve reusability. With this integration, dataflows can reference centralized variables using the Variable.Value and Variable.ValueOrDefault functions. This enables dynamic substitution of values, such as workspace or lakehouse IDs, based on environment-specific settings, eliminating the need for hardcoded parameters. Key Benefits Centralized configuration: Manage variables in one place across Fabric workloads. Environment-aware dataflows: Seamlessly switch between dev, test, and prod environments. Improved CI/CD support: Simplify deployment pipelines with this new variable integration with dataflows. Designed to help teams build more maintainable and scalable data solutions in Microsoft Fabric. Try it out and share your feedback with us! To learn more, refer to the Use Fabric Variable libraries in Dataflow Gen2 (Preview) documentation. Parameterized Dataflow Gen2 using public parameters mode (Generally Available) This powerful feature enables dynamic and flexible dataflow refresh by allowing parameter values to be passed externally—via the Fabric REST API or native Fabric experiences—without modifying the dataflow itself. Updates and Improvements Over the past few months in preview, we’ve introduced several enhancements based on user feedback: New Parameters Section in Recent Runs: easily view the exact parameters and values used during each Dataflow refresh. Improved Error Messaging: clearer diagnostics when refreshes fail due to missing or mismatched parameters. Expanded Data Type Support: use the following additional types in Public Parameters mode: Date DateTime DateTimeZone Time Duration This release marks a major step forward in making Dataflows more dynamic, reusable, and CI/CD-friendly. To learn more about this feature, refer to the Use public parameters in Dataflow Gen2 (Preview) documentation. New Discover Dataflow Gen2 Parameters API (Preview) The new Discover Dataflow Gen2 Parameters API in Microsoft Fabric empowers developers and data professionals to programmatically retrieve all parameters defined within a Dataflow Gen2 with CI/CD that has the public parameters mode enabled. This capability is a key part of the broader public parameters initiative, which aims to make Dataflows more transparent, reusable, and automation friendly. With a simple GET request to the endpoint users can access metadata about each parameter—its name, type (e.g., String, Boolean, DateTime), default value, and whether it’s required. Why it matters This API unlocks new possibilities for automation, governance, and integration. Developers can dynamically inspect parameter configurations before executing a Dataflow, ensuring compatibility and reducing runtime errors. It also simplifies building custom tooling and dashboards that visualize or validate parameter usage across workspaces. How it fits into the bigger picture The public parameters feature in Dataflow Gen2 is designed to make parameter definitions accessible and manageable across environments. By exposing parameters via REST, Microsoft Fabric enables seamless integration with CI/CD pipelines, monitoring tools, and external orchestration systems. Whether you're building enterprise-grade data solutions or lightweight automation scripts, the Discover Dataflow Parameters API is a foundational step toward more intelligent and scalable data operations. Dataflow Gen2 Support for Incremental refresh to Lakehouse as a data destination (Generally Available) With this update, you can now leverage incremental refresh to efficiently manage and update large datasets within your Lakehouse environments. This unlocks faster processing times and optimized resource utilization. This new capability empowers organizations to keep their Lakehouse data fresh while minimizing overhead and maintaining robust performance at scale. We look forward to seeing how this enhancement helps you streamline operations and deliver up-to-date insights with ease. Learn more about incremental refresh in Dataflow Gen2 in the Incremental refresh in Dataflow Gen2 documentation. New Dataflow Gen2 data destinations (Generally Available) New data destinations for Dataflow Gen2 are being introduced. Users can now write data directly to Lakehouse files (CSV) further expanding integration options for diverse analytics needs. Additionally, we're offering an early sneak peek at Snowflake as a destination, while it's entering preview. This glimpse gives you an opportunity to explore what's ahead and plan how best to leverage Snowflake within your dataflows when it becomes generally available. Whether you're working with teams on Fabric or on the M365 platform, Dataflow Gen2 allows you to collaborate with everyone. This milestone reflects our commitment to supporting enterprise-grade data management for a wide range of use cases. Finally, Incremental refresh support for Lakehouse tables in Dataflow Gen2 is now also generally available. This enables users to efficiently manage and update large datasets with faster processing and optimized resource utilization, ensuring your Lakehouse data remains fresh with minimal overhead. We look forward to seeing how these enhancements help you streamline operations and deliver timely insights across your data estate. Learn more about the new destinations in the Dataflow Gen2 data destinations and managed settings documentation. Schema Support in Dataflow Gen2 Destinations: Lakehouse, Fabric SQL and Warehouse (Preview) This powerful new capability makes it even easier to organize and manage your data within complex analytics environments. To take advantage of schema support, simply enable ‘Navigate using full hierarchy’ under Advanced settings when configuring the connection for your chosen data destination. With this enhancement, you can seamlessly map your data to the right schema, ensuring greater flexibility and alignment with your organizational standards. Explore this feature and let us know how it helps you optimize your data workflows. Learn more about schema support here in our advanced section in our documentation Dataflow Gen2 data destinations and managed settings. Dataflow Gen2 - Natural language to custom column with Copilot (Generally Available) Earlier this year we released a new copilot-driven experience that aims to help you in creating custom columns with just natural language. This experience is available within the 'Custom column' dialog which you can find through the 'Add column' tab of the ribbon. With it you can describe with a simple prompt what you wish your new column to calculate, and Copilot will create the formula for you. Give it a try today within Dataflow Gen2 and let us know your experiences with it. To learn more about this feature, refer to the Add a custom column documentation. Dataflow Gen2 – Explain query and query steps using Copilot (Generally Available) This new capability brings the power of AI directly into your data transformation workflows by helping you interpret Mashup (Power Query M) code in natural language. Whether you're reviewing a full query or a specific step, Copilot makes it easier than ever to understand and debug your dataflows. Key Features Explain this query: Triggered via the Copilot pane or by right-clicking in the Queries pane. Explain this step: Available by right clicking any step in the applied steps section of a query. This feature is designed to empower data professionals of all levels to work more confidently and efficiently with Power Query M code. To learn more about this feature, refer to the Copilot explainer skill in Dataflow Gen2 documentation. Copilot in Modern Get Data (MGD) for Dataflow Gen2 (Preview) With this new Copilot in MGD experience in Fabric Dataflow Gen2, you can ingest and transform data effortlessly with natural language commands. Discover a faster, smarter way to get the data you need. In Fabric Dataflow Gen2, select Get data to begin. In the Get data wizard, select the Copilot tab, then you can start with the list of recently used tables. After loading the recently used table, you can chat with Copilot to find the data you want. For step-by-step exploration, we want to first group by the data on customers’ titles to check the results. Then depending on the range of the counts, we can decide to include which ranges. When selecting table columns, use @ to quickly view available columns. Then entering the letter can filter on detail column. If you know all the operations you want to do in the beginning, you can describe all in one sentence. Then Copilot can quickly understand it and provide the filtered results to you. To return to the previous step, select the 'Restore' button next to it and your data will revert to that point. You can also Copy the preview data to confirm with your colleagues before saving it into Dataflow Gen2. Copilot in MGD offers transformation functions like Dataflow Gen2 Copilot. For details, refer to the Dataflow Gen2 Copilot documentation. Dataflow Gen2 – Postgre SQL Connector adds support for Microsoft Entra ID Authentication You can utilize Microsoft Entra ID authentication to connect to PostgreSQL databases in Dataflow Gen2, as an alternative to traditional username and password authentication. For more information, refer to the PostgreSQL connector documentation. 2-Tier Pricing Model for Dataflow Gen2 (CI/CD A new 2-tier pricing model for Dataflow Gen2 has been introduced, with the aim of making query evaluation more affordable and transparent for users. This update is part of our ongoing commitment to respond directly to customer feedback and deliver more cost-effective solutions for diverse workloads. First 10 Minutes: Query evaluation is now billed at just 12 CU—a 25% reduction from previous rates. Beyond 10 Minutes: For longer-running queries, costs drop dramatically to 1.5 CU—a 90% reduction, making extended operations significantly more budget-friendly. This pricing model is effective immediately for Dataflow Gen2 (CI/CD) operations. To take advantage of the new rates, users should upgrade any non-CI/CD items by using the ‘Save as Dataflow Gen2 (CI/CD)’ feature. For further details on this update, please refer to the Dataflow Gen2 Pricing documentation article. Pipelines Microsoft Fabric Data Factory Data pipelines are now ‘pipelines’ Why the Change? We’re making Fabric Data Factory pipelines more inclusive and adaptable by simplifying language and broadening our mission. ‘Data pipelines’ are now simply ‘pipelines’, reflecting our commitment to extending Fabric Data Factory’s capabilities to fit more diverse and extended use cases. This update isn’t just semantic—it signals a broadened horizon where pipelines can orchestrate not just data, but also services, applications, and business processes. By embracing a more unified terminology, we invite our community to imagine new possibilities, integrating data engineering with broader business and workflow automation in Fabric Data Factory. Locations in the UI reflecting the change Workspace Artifact Type Creating New Items Workloads in Data Factory OneLake Catalog Lineage view New Activities in Email and Teams (Generally Available) These updates introduce a modern UX and enhanced functionality—making it easier than ever to integrate communication seamlessly into your pipelines. What’s New? The new activities are designed to be more intuitive, flexible, and future-ready. Whether you're sending notifications, triggering actions, or collaborating across Teams, these activities streamline how you connect communication with orchestration. Modern UX: A refreshed interface that aligns with Fabric’s design principles. DMTS Support: Seamless integration with Data Movement and Transformation Services. Improved Scenarios: Enhanced support for common use cases like approvals, alerts, and status updates. When using user auth and deploying a pipeline with the Email or Teams activity to another workspace, if you are not the user who created the activity in the source workspace, the target workspace will have the activity set to inactive until you create a new user auth connection in the target workspace. Legacy Activities Get a Facelift We’ve also updated the UX for legacy activities to help users distinguish between old and new experiences. You’ll notice clearer labels and visual cues: Office 365 Outlook (Legacy) Teams (Legacy) Microsoft Teams Office 365 Email These legacy activities will remain available for a limited time, but we strongly encourage users to begin transitioning to the new versions. To learn more, check out our documentation on the Office 365 Outlook activity and the Teams activity. Dataflow activity in pipelines: New parameters experience Introducing a major usability upgrade to the Dataflow activity in Microsoft Fabric! When using a Dataflow Gen2 with public parameters mode enabled, you’ll now benefit from a streamlined experience that makes working with parameters faster and more intuitive. Thanks to the new Discover Dataflow Gen2 Parameters API, the Dataflow activity can now automatically detect and display all available parameters for the selected Dataflow. Data types Default values No more manual entry or guesswork—just select your Dataflow and immediately see what’s available. This enhancement helps you configure your pipelines more efficiently and with greater confidence. Try it out and experience the productivity boost firsthand! To learn more about this feature, refer to the Dataflow activity documentation. Debug your pipeline Expression with the Evaluate Expression experience The Expression Builder Evaluate experience is now available in Microsoft Fabric pipelines! This new capability is designed to make working with dynamic pipeline content easier, faster, and more intuitive. What is it? We’ve heard from many of you that writing and debugging expressions can be frustrating. The evaluate expression experience helps alleviate some of these problems by: Parsing expressions and showing how they resolve. Auto-populating default values for parameters and variables. Allowing manual input for runtime-specific values. Providing schema previews. The evaluate expression feature introduces a simple button that lets you test your pipeline expressions instantly. This tool helps you understand exactly how your expression will behave without needing to run the entire pipeline. To learn more, check out the documentation for Evaluate pipeline. Functions Activity with User Data Functions (Generally Available) This milestone unlocks powerful new capabilities for building reusable, secure, and parameterized logic within your pipelines. Easily invoke custom functions across multiple pipelines to streamline complex workflows. You’ll also be able to pass secure inputs and outputs, configure retry logic, and manage parameters with full control. With the integrated experience, you can select your workspace, function item, and parameters directly from the pipeline canvas. To learn more about setting up a Fabric User Data Function, check out this article on Create a Fabric User data functions item. Or the blog post on Utilize User Data Functions in Data pipelines with the Functions activity. Add up to 20 schedules for your pipeline! You now have the capability to add multiple schedules to your pipelines in Fabric Data Factory! With this new update, you can add up to 20 schedules to your pipeline, giving you greater flexibility to automate runs at different times. This enhancement allows teams to better align their data processing with organizational needs, ensuring timely data availability and improved operational efficiency. On-premises and VNet data gateway support for Invoke pipeline and Semantic model refresh activities Expanded secure connectivity options are now available in Microsoft Fabric Data Factory! With support for On-premises Data Gateway (OPDG) and Virtual Network (VNet) Data Gateway, you can orchestrate two of the most popular activities—Invoke pipeline and Semantic Model Refresh—while keeping your data secure inside your corporate network. Why This Matters Cross-Workspace pipeline Orchestration: From your Fabric pipeline, you can now invoke pipelines hosted in other workspaces—whether in Fabric, Azure Data Factory (ADF), or Synapse—even when those workspaces are secured with Managed VNets. This enables secure connectivity to on-premises or network-isolated resources without exposing any public endpoints. Semantic Model Refresh: Leverage the Semantic Refresh activity to refresh models within Fabric workspaces that are secured with Managed VNets, ensuring secure and compliant data access Parity with Azure Data Factory: Enjoy familiar patterns and security best practices, now available in Fabric. Prerequisites To get started, you’ll need: A Fabric workspace with permissions to create and use connections. An OPDG or VNet data gateway installed and configured by your tenant admins or networking team, and visible to your workspace. Setup Resources Install an on-premises data gateway Create virtual network (VNet) data gateways How to Use Gateway Connections 1. Invoke pipeline Activity In Fabric, open your pipeline and add the Invoke Pipeline activity. You can invoke another Fabric pipeline, Azure Data Factory pipeline, or Synapse pipeline. Choose Your Data Gateway: Under the Connection settings select your OPDG or VNet gateway, connect, and run! 2. Semantic model refresh Similarly, when refreshing a semantic model, you can select your OPDG or VNet gateway connection to securely access private data sources. Ready to try it? Set up your gateway, configure your connections, and start orchestrating secure data activities in Fabric today! Copy job Activity (Preview) This new orchestration activity simplifies data movement by bringing the familiar Copy job item directly into Microsoft Fabric Data Factory pipelines. You can now manage data transfers alongside transformations, notifications, and more—all within a single, unified experience. The Copy job activity includes a monitoring link that gives you real-time visibility into your Copy job progress and status. You can track execution outcomes, monitor performance, and quickly identify issues. To learn more about the new Copy job activity, check out our documentation on Copy job Activity in Data Factory pipelines Invoke pipeline activity (Generally Available) This release marks a major milestone, empowering users to seamlessly trigger and orchestrate pipelines within their Fabric Data Factory workflows, making it easier than ever to build robust and flexible data solutions. For more information, refer to the Invoke pipeline activity documentation Support for Workspace Identity Introducing support for Workspace Identity across key Microsoft Fabric Data Factory activities! With this update, you can now leverage workspace identity to securely and seamlessly execute Invoke Pipeline, Semantic Model, and Scope Activity operations. This enhancement simplifies authentication, strengthens security, and streamlines management by allowing you to use a unified identity when orchestrating data workflows. Workspace identity support reduces the need for manual credential management, making it easier to build and maintain robust pipelines that span multiple Fabric services. Whether you're invoking complex pipelines, accessing semantic models, or setting up scope activities, Workspace Identity ensures consistent access control and governance throughout your environment. Variable Library integration with pipelines (Generally Available) With this release, users can now seamlessly manage and reuse variables across multiple pipeline activities, simplifying workflow design and enhancing flexibility. The integrated Variable Library empowers teams to standardize variable usage, reduce errors, and streamline pipeline configuration, making it easier than ever to build scalable, maintainable data solutions. Azure Databricks Jobs activity (Generally Available) Databrick Jobs allow you to schedule and orchestrate a task or multiple tasks in a workflow in your Databricks workspace. Since any operation in Databricks can be a task, this means you can now run anything in Databricks via Fabric Data Factory, such as serverless jobs, SQL tasks, Delta Live Tables, and more. You can find the Job type in your Azure Databricks activity under the Settings tab. To learn more, check out the Azure Databricks activity documentation or the Orchestrate your Databricks Jobs with Fabric Data pipelines! demo. Apache Airflow Job New Fabric Airflow features make it easy to build DAGs Fabric Data Factory orchestration capabilities focus on more than just scaled-out low-code pipelines, which we’ve offered for many years in ADF & Fabric Data Factory. We also encourage code-first Python-based DAG orchestration by using Apache Airflow Job in Fabric Data Factory. Fabric Notebooks now seamlessly integrate with your Airflow DAGs, enabling enhanced collaboration, exploration, and automation. With just a few clicks, users can now easily embed Python code to call Fabric Notebooks directly within their Airflow workflows, leveraging rich data exploration and transformation capabilities right where orchestration happens. Start by selecting the ‘Add Connection’ helper button on the toolbar inside our built-in DAG editor to add your SPN connection to your Notebooks. Fabric Apache Airflow Job now supports CI/CD This new capability empowers data engineers and developers to seamlessly automate deployment and management of Airflow workflows, integrating best DevOps practices directly within the Fabric environment. In Fabric, you have two tools to support CI/CD: Git integration and deployment pipelines. Git integration lets you connect to your own repositories in Azure DevOps or GitHub. Deployment pipelines help you move updates between environments, so you only update what’s needed. With CI/CD support, you can enable faster iteration, maintain version control, and ensure reproducibility for your data orchestration pipelines—all while reducing manual effort and minimizing deployment risks. To learn more about this feature, refer to the CI/CD for Apache Airflow in Data Factory in Microsoft Fabric documentation. Copy job Data Replication from Fabric Lakehouse with Delta Change Data Feed (Preview) Copy job now supports Fabric Lakehouse Table connector with native CDC support. This connector enables efficient, automated replication of changed data—including inserts, updates, and deletes—from a Fabric Lakehouse via Delta Change Data Feed (CDF) to supported destinations. With this enhancement, your destination data stays continuously up to date—no manual refreshes, no extra effort. Making your data integration workflows more efficient and reliable. What this means Replicate data changes seamlessly from your Lakehouse after processing is completed in OneLake. Distribute updated data to supported destinations outside Fabric, such as SQL or Snowflake. Save time and reduce complexity with automated, change-aware data movement. This new CDC connector brings you the flexibility to keep downstream systems in multi-cloud environment in sync—ensuring your data is always accurate, timely, and ready for action. To learn more, refer to the Change data capture (CDC) in Copy job documentation. Connection Parameterization with Variables library for CI/CD (Preview) Copy job now supports connection parameterization via variable library! This powerful capability helps automate your CI/CD processes by externalizing connection values. With it, you can deploy the same Copy job across multiple environments while relying on the variable library to inject the correct connection for each stage. Meaning you can seamlessly use different data stores for development, testing, and production—without modifying your Copy job each time. Capabilities Parameterize connection using variables stored in the variable library in Fabric. Promote Copy job seamlessly across environments—for example, from Dev to Test to Production—without hardcoding or manually editing data store connections. Centralize configuration management, reducing duplication with a unified approach that makes it easier to manage configurations consistently across different environments. To learn more, refer to the CI/CD for Copy job in Data Factory documentation. Merge data into Snowflake You can now choose to merge changed data—including inserts, updates, and deletions—into Snowflake, when the data originates from any CDC source connectors such as Azure SQL DB, SQL Server, SQL MI, or Fabric Lakehouse tables. What’s more, with Storage Integration support in the Snowflake connector for Copy job, you gain enhanced security through a Snowflake-assigned role. This eliminates the need to expose sensitive credentials and allows you to implement more secure authentication methods when connecting to Azure Blob Storage. For more information, check out the following resources CREATE STORAGE INTEGRATION or Change data capture (CDC) in Copy job. More Connectors, More Possibilities More source and destination connections are now available, giving you greater flexibility for data ingestion with Copy job. We’re not stopping here—even more connectors are coming soon! Newly supported connectors Folder REST SAP Table SAP BW Open Hub Amazon RDS for Oracle Cassandra Greenplum Informix Microsoft Access database Presto Incremental copy now supported for more connectors SAP HANA MariaDB MySQL SFTP FTP Oracle cloud storage Amazon S3 Compatible Learn more from the What is Copy job in Data Factory documentation. Simplified Copy Assistant, Powered by Copy job Access the full power of the Copy job by selecting ‘Copy Assistant’ from a pipeline. This streamlined experience makes it easier to configure and manage data movement within your workflows. Eliminating the need for unnecessary parameterized foreach loops and copy activities as before for simple data copying. It also empowers you to benefit from all Copy job capabilities, including native incremental copy and change data capture (CDC). Learn more from the What is Copy job in Data Factory documentation. Connectivity New connectors available in Data Factory The addition of a wide range of new connectors in Fabric Data Factory expands the options available for various data integration scenarios. These new connectors that are now supported across Copy job, Copy activity, and Lookup activity in Pipeline, giving you even greater flexibility to connect with diverse data sources. Generally Available Amazon RDS for Oracle (Bring your own driver) Cassandra Greenplum HDFS Informix Microsoft Access database Presto With these additions, Dataflow Gen2 continues to grow as a powerful data preparation and transformation engine, designed to meet the needs of modern data integration. By expanding the connector portfolio, we’re giving you the flexibility to seamlessly connect to the tools and platforms your business depends on, while ensuring performance, reliability and security. Generally Available Snowflake 2.0 Preview Google BigQuery 2.0 Impala 2.0 Netezza (Bring your own driver) Vertica (Bring your own driver) Oracle (built-in driver) – OPDG only Learn more about the connector availability across Data Factory in the Connector overview documentation. Learn more about the connector availability across Data Factory in the Connector overview documentation. Salesforce & Salesforce Service Cloud connector: Partition auto detection We are dedicated to enabling organizations to realize the complete potential of their data by providing integration solutions that are seamless, dependable, and scalable. To keep pace, we continuously ship connector innovations to simplify the developer’s experience and improve productivity. Now, both Salesforce and Salesforce Service Cloud connectors now support reading data with partitions. This enables users to pull data from Salesforce tables using multiple threads for significantly improved performance. Best of all, there’s no need to manually configure partition details. The connector intelligently detects and applies the optimal partitioning strategy. This update greatly simplifies the integration experience while delivering faster throughput. Available as an advanced setting in the Salesforce connector for copy activity, we highly recommend leveraging this feature for long-running copy tasks that can benefit from multi-threaded reads. Learn more details about this feature in the Partition option settings for Salesforce Connector documentation. Support upsert to delta table with Lakehouse connector (Preview) Fabric Lakehouse connector is one of the core connectors that empowers enterprise to bring data into Fabric OneLake. We’re continuously delivering innovations to simplify data ingestion. The latest enhancement adds upsert support to the Lakehouse connector, allowing you to write directly to Delta tables. With upsert, you can efficiently manage incremental data loads and maintain consistency—without relying on complex workarounds. This powerful capability is available in both Copy job and Copy activity within Pipeline. Learn more about this feature in the Table action settings from Lakehouse Connector documentation. Support delta column mapping and deletion vector with Lakehouse connector The Lakehouse connector now supports delta column mapping and deletion vectors, further strengthening its ability to work seamlessly with delta tables in Fabric OneLake. With column mapping, you gain flexibility to handle schema evolution and align columns across different systems without manual adjustments. Deletion vector support ensures that data operations remain accurate and consistent, even when rows are deleted or updated. These enhancements reflect our ongoing commitment to evolve the Lakehouse connector, with more delta table capabilities to help you manage and analyze your data even more effectively. Learn more details about this feature in the Lakehouse connector documentation. Support varchar(max) in table creation with Data Warehouse connector (Preview) The Fabric Data Warehouse connector now supports the varchar(max) data type during table creation, giving you greater flexibility to handle large text data without truncation. Users can specify this data type in column mapping when creating tables, whether using Copy job or Copy activity in Pipeline. Making it easier to ingest and store extensive text fields such as detailed logs, descriptions, or notes directly into your warehouse, streamlining data integration for real-world scenarios. Learn more details about this feature in the Data warehouse connector documentation. DB2 connector: support to specify for package collection The DB2 connector now allows users to specify a package collection directly within the pipeline, providing greater control and ensuring better alignment with DB2 database configurations. This capability is available as an advanced setting in copy activity, enabling more precise and efficient data integration. Learn more details about this feature in the DB2 connector documentation. Snowflake connector: support to specify for role The Snowflake connector now supports specifying a role directly within your pipeline. This enables users to run the copy with right level of access, helping stay aligned with the organization’s security and governance practices without extra configuration steps. This is now an advanced setting available in copy activity. Learn more details about this feature in the Snowflake connector documentation. Gateways Virtual Network Data Gateway supports Fabric pipeline and Copy job (Generally Available) You can now use Pipeline and Copy job with the Virtual Network (VNET) Data Gateway in Microsoft Fabric. Enabling a secure, high-performance way to move data between private networks and Fabric without exposing it to the public internet. This integration is especially valuable for industries with strict compliance requirements—such as finance, healthcare, and government. It ensures data remains within private network boundaries, eliminating the complexity of VPNs and reducing the risks associated with public endpoints. With end-to-end security, faster transfers via private endpoints or ExpressRoute, and a simple setup that works seamlessly with existing Pipeline and Copy job configurations, you can move your data with confidence, compliance, and speed. To learn more, refer to the Use virtual network data gateway with pipeline in Fabric documentation. Mirroring Mirroring for Google BigQuery (Preview) This new capability allows customers to continuously replicate BigQuery data into OneLake —Fabric’s Unified Data Lake—with zero ETL. With near real-time replication and native integration across the Fabric experience, Mirroring makes it seamless to bring BigQuery data into Fabric. This unlocks the full power of Microsoft’s integrated suite for analytics, processing, and reporting—enabling you to derive insights faster and more efficiently than ever before. To learn more and to get started, please reference the Mirroring for Google BigQuery (Preview). Mirroring for Oracle (Preview) This new capability empowers customers to continuously replicate data from their Oracle databases—including on-premises, Oracle OCI, and Exadata—directly into OneLake, Fabric’s unified data lake, with near real-time performance and zero ETL. Mirroring for Oracle is natively integrated across the Fabric experience, making it seamless to bring Oracle data into Fabric. With rapid replication and instant availability in the data warehouse view, you can unlock the full power of Microsoft’s analytics, processing, and reporting suite—enabling faster, more efficient insights from your Oracle data. Supported environments include Oracle on-premises, Oracle OCI, and Exadata. The setup is straightforward, with guidance available for enabling necessary database configurations and permissions. To learn more and to get started, please reference our blog post on Mirroring for Oracle (Preview). Mirroring for Azure SQL Managed Instance (Generally Available) This milestone marks a significant step forward in our mission to provide seamless, near-real-time data replication capabilities, empowering you to derive maximum value from your SQL data with Microsoft’s unified data platform. Mirroring for Azure SQL Managed Instance offers continuous data replication into OneLake, ensuring that your data remains current and readily accessible for advanced analytics and reporting needs without complex ETL processes. To learn more and to get started, reference Mirroring Azure SQL Managed Instance (Preview) documentation. Mirroring support for sources behind a firewall (VNET and OPDG) With this release, organizations can now securely and efficiently replicate data from key sources – with general availability for Snowflake, Azure SQL Database, and Azure SQL Managed Instance – using either the On-Premises Data Gateway (OPDG) or the VNET Data Gateway. This ensures seamless data movement into OneLake, while maintaining robust security and compliance for your most critical workloads. Whether your databases reside on-premises or within a virtual network, Mirroring in Fabric provides flexible connectivity options to enable encrypted, high-throughput, and low-latency connections – without exposing your data sources directly to the internet. This unlocks real-time analytics, reporting, and AI across your entire data estate. And this is just the beginning: every new source supported by Database Mirroring will also support these gateway options, so you can expect even broader coverage in upcoming releases. To learn more and to get started, reference the Mirroring Azure SQL Managed Instance (Preview) documentation. Azure SQL Database mirroring now supports Workspace Identity authentication You can now use Fabric Workspace Identity authentication to mirror your Azure SQL Database, in addition to the basic (username and password), organization account and service principal authentication options. Workspace Identity authentication in connections leverages Microsoft Entra ID to provide seamless, secure access to data sources using your Fabric workspace’s managed identity. This modern authentication approach eliminates the need for storing credentials while providing fine-grained access control and comprehensive audit capabilities. To learn more, refer to the Workspace identity and Tutorial: Configure Microsoft Fabric Mirrored Databases from Azure SQL Database documentation. Developer tooling Fabric VS Code extension (Generally Available) We’ve heard your feedback and have made several new features, and improvements to Fabric VS Code Extension. The extension offers enhanced capabilities for managing Fabric items, multi-workspace support, and direct integration with Fabric SQL databases—all within Visual Studio Code. Key features Programmatic Management: Manage Fabric items programmatically using item definitions, enabling scripting and file-based workflows. This makes it easier to edit in VS Code and publish your changes. Git integration: You can clone your Git enabled workspace using the extension and use VS Code’s source control experience to work with your items and push the changes to the repository. Note that only Azure DevOps is supported for now. Multi-Workspace Support: View and filter multiple Fabric workspaces simultaneously within the extension. You can easily switch tenants to work on workspaces and items across tenants. SQL Database Integration: Directly open and work with Fabric SQL databases in VS Code with the SQL Server extension. Figure: View and manage your workspaces and item in VS Code Check out the documentation for What's new in Microsoft Fabric extension (Generally Available) to learn more. Extensibility Lucid Data Hub’s Fabric Workload (Preview) Lucid Data Hub has launched Agent Mart Studio as a workload in Microsoft Fabric, empowering business users to build and deploy AI agents directly on enterprise data in OneLake. This integration allows organizations to automate complex business processes with AI agents that leverage unique industry knowledge and are protected by contextual guardrails. The Lucid Data Hub + Microsoft Fabric: Empowering Business Users with AI Agents blog post highlights how Lucid’s no-code environment enables non-technical users to create and customize agents for real-time insights and workflow automation, such as retail out-of-stock detection. Check out the short Agent Mart Studio demo. Statsig’s Experimentation Analytics on the Fabric Workload Hub (Preview) Statsig Experimentation Analytics on Microsoft Fabric offers an integrated solution that empowers product teams to innovate and accelerate data-driven decision making by unifying experimentation, feature rollout, and impact analysis within the Microsoft Fabric ecosystem. It provides a frictionless, secure way to store and analyze experimentation data by providing a warehouse native experience with the ability to define custom metrics and support rigorous statistical testing, analysis and visualization of product and behavioral data stored in OneLake without the need for data movement or running complex ETL pipelines. Start using the workload Statsig’s Product page. Closing We hope that you enjoy the update! Be sure to join the conversation in the Fabric Community and check out the Fabric documentation to get deeper into the technical details. As always, keep voting on Ideas to help us determine what to build next. We are looking forward to hearing from you!The Foundation for Powering AI-Driven Operations: Fabric Real-Time Intelligence
Every organization shares the same ambitions: to deliver better outcomes, increase efficiency, mitigate risks, and seize opportunities before they are lost. These ambitions underpin growth, resilience, agility, and lasting competitive advantage. Yet most organizations struggle to harness the full value of their data to realize those ambitions. Massive volumes of granular signals flow in constantly from warehouses, stores, customers, factories, vehicles, applications, and other entities, but most go unused. Hidden in this flood are the signals that matter most: the customer about to churn, the machine about to break down, the shipment about to be delayed. Too often they go undetected, or actions come too late to change the outcome. The barriers are well-known. Data arrives too late. Volumes are overwhelming. Batch and real-time systems remain siloed. Humans struggle to track the countless entities moving through operations and to isolate all the high-value signals and act in time to make a difference. This is where AI changes the equation. AI can process signals at scale, identify the ones that matter, and determine the right action in the moment. The question every leader must now confront is: how can AI transform how my organization operates to unlock real value and deliver outcomes? The age of Agentic AI: from reactive to proactive at scale We are now in the fourth great evolution of software. The first was client–server. The second was cloud. The third was software-as-a-service (SaaS). And today we are entering the fourth: agentic AI. Unlike the narrow applications of chatbots or content generation, the real revolution is unfolding in operations. With agentic AI, applications no longer stop at predefined reports, forms, and workflows. They act as intelligent systems that observe, reason, adapt, and act autonomously. Two types of AI make this shift possible. Analytical AI identifies the signals that matter by uncovering patterns, anomalies, and trends, explaining what happened and why. Agentic AI goes further, interpreting those signals, making decisions, and acting in time to change outcomes. The true transformation comes from combining the two: creating agents that not only detect what matters but also take the right timely action. But intelligence is only as good as the data it is built on. AI cannot operate in a vacuum. It needs up-to-date data and rich context: how events unfold across time and space, how systems interconnect, and how operations run. Without that foundation, their decisions will be incomplete or outdated. For too long, operations have been reactive, analyzing data after the fact and responding when it was already too late, if at all. Now, leading businesses across industries are breaking the cycle: airlines dynamically adjust schedules, retailers respond instantly to demand shifts, manufacturers prevent costly downtime, hospitals improve patient flow in real time. They are turning signals into outcomes, moving from reactive firefighting to proactive, autonomous decision-making. Their advantage comes from one source: Real-Time Intelligence in Microsoft Fabric, delivering the fresh unified data, contextual understanding, and AI capabilities that make AI-powered operations a reality. Real-Time Intelligence in Microsoft Fabric Real-Time Intelligence in Microsoft Fabric makes it simple to build complete event-driven solutions: from capturing signals the moment they happen, to analyzing them in real time, to modeling assets and systems as digital twins, to visualizing insights, and to triggering alerts and actions instantly. It places every signal in context with the rest of your data in Fabric and powers AI agents with fresh intelligence. With its low- and no-code experiences, what once required advanced expertise and complex integrations is now accessible to any team, forming the operational nervous system of the modern enterprise. This simplicity does not come at the expense of scale and governance. Real-Time Intelligence runs on the same planet-scale infrastructure that powers Microsoft and Azure. Live data flows into OneLake as a unified, governed estate, seamlessly combines with historical context, and becomes actionable in Power BI, Teams, Power Automate, and thousands of connected systems. Business users can build dashboards with Copilot and define triggers, while developers extend with APIs and pro-code tools. As a native Fabric workload, Real-Time Intelligence inherits shared governance, lineage, and security, ensuring real-time workloads scale safely across the enterprise. Today, Fabric Real-Time Intelligence delivers building blocks for rapidly creating complete, event-driven applications and provides the foundation for the shift to agentic AI: Eventstream: capture and transform streaming data from any source. Eventhouse: store and query petabytes of events at scale with lightning-fast performance. KQL Queryset: author and share reusable queries for real-time analysis. Real-Time Dashboard: visualize key signals and insights as they unfold. Copilot: use natural language to generate queries, uncover insights, and build dashboards. Digital Twin Builder: model assets, entities, and processes for operational intelligence. Activator: detect conditions in live data and trigger alerts or workflows. Real-Time Hub: discover and manage all your events in one place. Customer and partner momentum Real-Time Intelligence is the fastest-growing workload in Microsoft Fabric, with nearly half of Fabric’s 24,000+ customers already using it and adoption growing 6x in the past year. This momentum reflects a simple truth: organizations everywhere are under pressure to move from reactive operations to proactive, real-time intelligence, and they are turning to Real-Time Intelligence to make it happen. This platform is rapidly delivering transformative impact for both direct-to-consumer (DTC) innovators and business-to-business (B2B) enterprise leaders. It is powering seamless, event-driven operations whether it is real-time passenger flows, baggage tracking, omnichannel retail, hospital logistics, or supply chain automation, for organizations of every size and vertical. Direct-to-consumer pioneers are reimagining their customer experience with real-time data. In hospitality, Valamar is delighting guests with seamless multi-modal check-in and check-out across touchpoints. In aviation, Emirates is reimagining passenger flows in real time, while Avinor is enabling real-time baggage tracking to improve airport efficiency and passenger experience. In food delivery and quick commerce, Swiggy is processing countless orders every hour across India. As Madhusudan Rao, CTO of Swiggy, explains: “Fabric’s Real-Time Intelligence empowers us to analyze clickstream and operational data instantly, helping us detect anomalies, and keep our promise of speed to millions of customers.” For B2B enterprise innovators, the gains are just as profound. In retail, Iceland Foods is streamlining operations across both stores and supply chains. In healthcare, Apollo Hospitals is improving patient flow and care delivery.In manufacturing, Owens Corning is optimizing production at scale. In energy, Veolia is managing critical resources with greater efficiency.In distribution, Sonepar is enabling smarter, more responsive logistics for professional customers across 40 countries. And in pharmaceuticals, Fresenius is advancing quality and safety with real-time responsiveness. The_Foundation_for_Powering_AI-Driven_Operations_Fabric_Real-Time_Intelligence This dual-spectrum impact across consumer brands and enterprise leaders shows how Real-Time Intelligence adapts to the evolving needs of every organization in today’s AI era. And this is only the beginning. This momentum is amplified by a powerful partner ecosystem. More than 40,000 consultants worldwide have been trained on Real-Time Intelligence, supported by over 6,500 partner organizations and 95 Featured Partners delivering real-time solutions. Powering industrial and physical AI with Real-Time Intelligence As the digital world accelerates into the fourth era of software, the physical world must evolve just as quickly. Asset-heavy industries, such as manufacturing, energy, logistics, and others, cannot afford to be left behind. The business stakes are high: operational efficiency, resilience, and returns on capital now depend on the ability to sense and act in real time. Leaders are already demonstrating what is possible with Fabric Real-Time Intelligence. Owens Corning is optimizing production at scale. Veolia is managing resources with greater efficiency. Hanwha is shifting from manufacturing-centric operations to service-led business models. And Sonepar is transforming supply chain responsiveness across 40 countries, meeting rising customer expectations with instant, data-driven action. Yann Shah, VP of Data, Analytics & AI at Sonepar, underscores the urgency: “At Sonepar, real-time capabilities are essential. Our professional customers, spanning 40 countries, demand immediate responses, and our e-commerce platform must meet these expectations. With Microsoft Fabric and Real-Time Intelligence, we convert live data into swift actions.” The opportunity goes far beyond asset monitoring and supply chain responsiveness. The next era of industrial AI will be defined by digital twins, semantic understanding, and agentic AI working together to make operations proactive and autonomous. Microsoft and NVIDIA are collaborating to accelerate the next era of industrial and physical AI. The integration of NVIDIA Omniverse™ libraries and OpenUSD with Microsoft Fabric Real-Time Intelligence brings real-time operational insights, agentic AI, and physically accurate digital twins to industrial operations. Sight Machine, a developer of industrial AI solutions, is leveraging Fabric Real-Time Intelligence and NVIDIA Omniverse™ libraries technologies to build advanced AI-powered digital twins for the automotive, pharmaceutical, and process industries. These solutions empower manufacturers to unlock new possibilities in observability and bring greater intelligence and automation to operations. Learn more from their blog. Introducing new innovations to power AI-driven operations Real-Time Intelligence is already transforming how organizations operate, but we are not stopping there. To truly power AI-driven operations, organizations need more than fast pipelines and dashboards. They need a complete understanding of operations, not just when events happen, but also where they occur and how they are related. They need the ability to rapidly identify the signals that matter in a sea of data. And they need the data to power AI agents that can autonomously act in time to make a difference. The_Foundation_for_Powering_AI-Driven_Operations_Fabric_Real-Time_Intelligence That is why today we are thrilled to announce major innovations in Microsoft Fabric Real-Time Intelligence: Maps, Graph, and Anomaly Detector. Together, these capabilities expand Real-Time Intelligence into a comprehensive platform for transforming operations, making intelligence more contextual, more actionable, and more autonomous. Map: Adding geospatial reasoning to intelligence Real-Time Intelligence has always started with when events happen. But in the real world, operations often depend on where they happen. Location is often a missing but critical dimension. With the new Map item, geospatial visualization and reasoning is as simple and intuitive as any other Fabric capability. Using intuitive no-code experiences, business users can create, customize, and consume map-centric applications bound to live data in Eventhouse, Lakehouse, and OneLake. Multiple layers from different sources can be overlaid on a single map, making it possible to see assets, people, and events as they move, all without GIS expertise or developer skills. Maps in Fabric transforms massive volumes of location-based data into interactive, always-up-to-date map visuals designed for quick exploration. The impact is immediate: organizations can monitor the flow of goods across global supply chains, watch sensor data streaming from infrastructure assets, or visualize customer activity across retail networks in real time. Because the data flows continuously, Maps delivers a live geospatial view of operations, enabling sharper awareness and smarter, location-aware decisions. And this is just the beginning. By bringing the where alongside the when, Maps sets the stage for the next wave of AI-driven operations, where intelligent systems can not only track what is happening, but anticipate what is coming, optimize logistics and routing, and identify the best course of action. Graph: Understanding relationships and dependencies Operations are not linear, they are networks. Suppliers link to distributors, machines connect to production lines, customers interact across journeys, and assets tie into broader systems. Success depends not just on what happens, but on how entities connect and influence one another. The new Graph capability in Fabric brings relational and causal reasoning directly into operations. With simple low- and no-code tools and natural language querying, users can build graph models in minutes, exploring connections and multi-hop dependencies across customers, assets, suppliers, and systems. Graph in Fabric is built upon the proven architecture principles of LinkedIn’s graph technology, one of the largest and most heavily used production graphs in the world. It delivers a scale-out architecture, built-in graph algorithms, and intuitive visual exploration that serve everyone from business users to data scientists. The value is immediate, organizations can detect weak links in supply chains, uncover cascading risks across systems, or see how disruption in one plant impacts production downstream. Customer journeys can be mapped more clearly, showing how one interaction shapes the next. Graph provides the relational context that turns isolated signals into connected intelligence. By weaving the how into the when and where, Graph opens the door to AI-driven operations that not only map relationships, but simulate outcomes, anticipate cascading effects, and trigger interventions before risks spread or opportunities are lost. Anomaly Detector: Uncovering the signals that matter Hidden in that sea of signals are the early warnings of risks and the fleeting opportunities that could change outcomes. The challenge has always been the same: how do you spot them in time and separate true anomalies from ordinary patterns? The new Anomaly Detector item is designed to do exactly that. Continuously scanning both real-time and historical data, it surfaces unusual patterns that would otherwise go unnoticed. By learning from historical behavior, the detector automatically builds a baseline of what “normal” looks like. That context is critical. Without it, anomalies in the moment cannot be distinguished from ordinary fluctuations. Historical training ensures the system recognizes seasonality, cyclical trends, and operational rhythms, making real-time detection sharper and more reliable. Using no-code experiences, business users can detect anomalies the moment they emerge, without needing data science expertise. The value is immediate, a sudden drop in production can be flagged before it impacts supply. A spike in customer churn can be caught before it spreads. Shifts in demand, anomalies in sensor readings, or irregularities in financial transactions can all be identified and acted on in the moment. Anomaly Detector helps detect many critical signals so that potential opportunities to act are not missed. By bringing Analytical AI directly into the heart of operational data, it enables organizations to move from simply recording what happened to proactively shaping what happens next. Additional innovations and enhancements Alongside the major announcements, we are also delighted to announce a wave of smaller but powerful enhancements in Real-Time Intelligence, the kinds of improvements that customers have been asking for and that make adoption easier, faster, and simpler. These updates span the entire Real-Time Intelligence experience. New Event Schema Sets and Microsoft streaming data source integrations like Azure Monitor diagnostic logs simplify data onboarding. Expanded private link support for Eventstream and Eventhouse strengthens security and compliance for sensitive workloads. New connectors (including MongoDB for Eventstream) and expanded Activator triggers (such as Spark Jobs and Fabric User Data Functions) make it easier to integrate intelligence into existing workflows. And with Copilot-powered dashboard exploration, teams can analyze signals more intuitively than ever. Each of these enhancements might seem incremental on its own. But together, they reflect our commitment to listening to customers and continually removing barriers. They make Real-Time Intelligence not just a platform for breakthrough innovation, but also one that evolves through steady, practical improvements that matter day to day. You can explore the full list of upcoming features in the product roadmap. The unified and complete solution for AI-powered operations With these innovations, Real-Time Intelligence is no longer just about speed. It is about a complete, 360-degree understanding of your operations. Organizations now have time, space, and relationships woven into a single operational fabric. They can move seamlessly from insight to action, having humans and AI agents working side-by-side to run operations smarter, faster, and more autonomously. The introduction of Map, Graph, and Anomaly Detector builds on a strong foundation that already includes Eventstream, Eventhouse, Digital Twin Builder, Activator, Real-Time Dashboard, and Copilot. Together, they form a unified, highly integrated platform, one that turns fragmented, complex, and costly real-time systems into a single, accessible solution. And because it is built with no-code and low-code tools, this transformation does not require armies of specialists or prohibitive upfront investment. It is intelligence that is usable by everyone. The implications are profound, the gap between leaders and laggards is widening. Leaders are already building real-time, AI-powered operations as a competitive moat, capturing the signals that matter, understanding them in context, and acting before the moment is lost. The shift is not theoretical, it's underway. The only question is: who will lead it? With Microsoft Fabric Real-Time Intelligence, you can transform your operations today and secure your place in the era of agentic AI. The future of operations is real-time and AI-powered, and the future is here. . Explore additional resources on Real-Time Intelligence Engage Submit ideas and vote: Fabric Ideas Ask questions on the: Real-Time Intelligence Forum Stay updated Microsoft Fabric Blog Microsoft Fabric YouTube Channel Follow on LinkedIn: Follow Real-Time on LinkedIn Check the Microsoft Fabric Release Plan Learn more Real-Time Intelligence Documentation Graph Documentation Complete the learning path: Implement a Real-Time Intelligence solution Get certified: Real-Time Learning Path Read the free eBook: The Democratic Data Revolution: Real-Time Insights for All Find customer success stories: Fabric Customer Success Play the Kusto Detective Agency Fabric edition: Kusto Detective Agency | Solve Data Investigation Challenges Complete the tutorial: Real-Time Intelligence Tutorial Try an extensive tutorial: FabCon RTI Tutorial Sign up for one-day training: Real-Time Intelligence in a Day
From insight to action: Bringing Fabric Activator into Ontology with Rules
With the introduction of Rules in Ontology, Fabric IQ takes a step forward in connecting business operations to real-time action by integrating Fabric Activator directly into Ontology. Fabric IQ brings context to your data. Activator in Fabric IQ operationalizes your ontology. What are Ontology Rules with Fabric Activator? Ontology Rules let you define conditions and actions on top of your business entities, rather than on raw tables or telemetry streams. These rules are evaluated using Fabric Activator, which monitors and triggers actions when conditions are met. The unique value is that the rule logic is expressed in the language of your business, using ontology entities and properties. Ontology defines what things mean (entities, relationships, context). Activator handles when something matters, what should happen. Why this matters Business logic, not technical plumbing Instead of hard-coding thresholds into pipelines or writing one-off stream queries, teams can define rules against business concepts like Customer, Order, or Device, making logic easier to understand, govern, and evolve. Consistent meaning across analytics, AI, and operations Because rules are grounded in Ontology, the same definitions used for AI agents like Fabric Data agents A Real-World Scenario: Cold-chain monitoring for retail operations A retailer models its business in Ontology with entities like: Store Freezer Product SaleEvent Using Ontology Rules, the team defines a rule such as “When a freezer’s temperature exceeds safe limits for a sustained period, trigger an email”. The rule is defined in business terms and enforced in real time. Figure: Add rule to an Entity type. Ontology Rules represent an important evolution of Fabric IQ from understanding the business to running it intelligently. This is foundational for the next wave of AI-powered operations, where agents and systems act based on what the business intends. You can read more about this at the rules documentation page. Learn more Check out Fabric IQ sizzle video to see it in action. Get an overview with Fabric IQ. Get started with Fabric IQ tutorial. Engage Submit ideas and vote. Ask questions on the forum.
