Monitor your Eventstreams with Workspace Monitoring — Now with per-Eventstream control (Preview)
When a real-time data pipeline breaks, the first question is always the same: what went wrong, and where? Eventstream observability in Microsoft Fabric brings that visibility directly into Workspace Monitoring, so you can track the health, performance, and errors of your eventstreams without leaving the Fabric portal or setting up external monitoring infrastructure. With this update, eventstream observability returns with a redesigned approach that puts you in control — including a per-eventstream toggle that lets you decide exactly which eventstreams emit monitoring data and which don't. Why observability matters for real-time pipelines Real-time data pipelines operate continuously. Unlike batch jobs that run and finish, an eventstream can process data for days or weeks without anyone checking on it. When something goes wrong — a destination write error, or backlogged events — the problem often goes unnoticed until downstream consumers report stale or missing data. Eventstream observability closes that gap. When enabled, each eventstream emits structured metrics and error data to your workspace's monitoring Eventhouse. This data lands in three dedicated KQL tables that you can query, alert on, and build dashboards against: Table What it captures Emission frequency When to use it EventStreamMetrics Throughput, event counts, ingress and egress rates ~1 minute Capacity planning, data flow validation, throughput monitoring EventStreamErrorMetrics Error category, error code, severity, affected events ~1 minute Troubleshooting failures, identifying recurring error patterns EventStreamNodeStatus Node state (running, stopped) ~6 hours Health checks, topology status overview Once monitoring is enabled for an eventstream, these tables appear in your workspace's monitoring KQL database alongside existing Eventhouse and other workload tables. Figure: Eventstream monitoring tables in the Workspace Monitoring KQL database, showing EventStreamMetrics, EventStreamErrorMetrics, and EventStreamNodeStatus tables available for querying. What changed: Per-eventstream control During the initial preview, eventstream observability was enabled globally for all eventstreams in a workspace. Based on customer feedback, the feature now provides granular control — you choose exactly which eventstreams emit monitoring data, giving you flexibility to focus observability on the pipelines that matter most. The key changes: Per-eventstream toggle. A new setting — Log Eventstream activity — appears in each eventstream's settings panel. You can enable or disable observability for each eventstream individually, so a workspace with ten eventstreams can monitor only the two or three that are most critical. Default OFF. All eventstreams default to monitoring OFF. You choose which ones to enable. How to get started Eventstream observability requires Workspace Monitoring to be enabled on your workspace. If you haven't set it up yet: Open your workspace settings. Navigate to the Monitoring section. Enable Workspace Monitoring and wait for the monitoring Eventhouse to provision. Once Workspace Monitoring is active: Open any eventstream in your workspace. Select the Settings gear icon. Find the Log Eventstream activity toggle. Enable it. Within a few minutes, data begins flowing into the eventstream tables in your monitoring KQL database. You can query them directly with KQL, build Real-Time Dashboards, or use KQL querysets for deeper analysis. What comes next This release covers performance metrics, error metrics, and node health — the foundation for operational observability. Additional capabilities are planned for future updates: Diagnostic logs — Error logs from the ASA processing engine, providing deeper root-cause analysis for failures like deserialization errors, runtime query errors, and output write failures. Expanded connector coverage — Monitoring support for connector-based sources is under investigation and will be added in a future phase. Next steps Refer to the Eventstream monitoring in Workspace Monitoring documentation to learn more about configuration and supported tables. Refer to the Workspace Monitoring overview documentation for setup instructions. Share your feedback and feature requests in the Fabric Community Forums. Explore the Create a Real-Time Dashboard documentation to build live dashboards from your monitoring data.
Fabric Runtime 2.0 (Generally Available)
Fabric Runtime 2.0 (Generally Available), marks a major milestone for Microsoft Fabric. As the execution foundation for Microsoft Fabric's Data Engineering and Data Science experiences, Runtime 2.0 delivers a modern, high-performance platform built on Apache Spark and deeply integrated across the Fabric ecosystem.
Change Event Streaming for SQL database in Microsoft Fabric (Preview)
SQL database in Microsoft Fabric already brings operational and analytical workloads together—now it can stream changes as they happen. Change Event Streaming sends inserts, updates, and deletes directly to Fabric Eventstream or Azure Event Hubs. Build near-real-time pipelines from operational data without leaving Fabric.Automate recurring business logic with scheduled User Data Functions in Microsoft Fabric (Preview)
Many business processes rely on custom code that must run repeatedly—whether to validate data, apply business rules, synchronize systems, or generate notifications. Until now, teams often needed separate orchestration tools or manual processes to keep those workflows running. User Data Functions in Microsoft Fabric let developers encapsulate custom business logic and run it close to their data. With scheduling support through the Fabric job scheduler, teams can run those functions automatically on a recurring cadence instead of relying on manual execution or separate orchestration services. This helps teams automate repeatable work such as validating data, processing feedback, generating business events, synchronizing external data, and sending operational notifications. The result is a simpler automation model that stays inside the Fabric workspace and uses familiar Fabric experiences for setup, monitoring, and troubleshooting. Figure: Create a schedule directly from the User Data Function experience, including execution frequency and parameter values. Why scheduling matters Teams often rely on separate orchestration workflows or manual processes to run recurring business logic. Scheduled User Data Functions reduce that complexity by letting teams attach a schedule directly to a function. The function remains the place where the business logic is defined, while the job scheduler handles recurring execution. The main benefits include: Simpler automation: Run recurring logic without building a separate orchestration workflow for every task. Workspace-centered operations: Configure, run, and monitor scheduled work from Fabric. Reusable logic: Use parameters to apply the same function to different inputs or scenarios. Operational visibility: Track execution history and outcomes through Fabric monitoring experiences. Faster issue detection: Use failure notifications to help teams respond when scheduled work fails What you can automate Scheduled User Data Functions are useful when the same logic needs to run repeatedly, and the logic is best expressed as code. They are especially helpful when the workflow is business-specific, parameterized, and tied to data already managed in Fabric. Common scenarios include: Run data quality checks before downstream reports or semantic models depend on the data. Analyze customer feedback at a regular cadence and write enriched results back to Lakehouse. Evaluate business rules and generate events when important changes occur. Synchronize data from external systems into Fabric-managed data stores. Send notifications when thresholds, exceptions, or operational conditions require attention. Archive stale records or run maintenance logic on a predictable schedule. These scenarios share a common pattern: the logic is repeatable, the timing matters, and the team benefits from keeping the automation close to the data. Scheduling helps teams operate that logic without creating a separate automation layer for every recurring task. Getting started with scheduling functions After the function is defined, you can configure a schedule from the function experience. The schedule allows you to define at what cadence these functions should run and what parameter values should be used. To learn more, refer to the Schedule a user data function documentation. Monitor scheduled runs Scheduled executions appear in Fabric monitoring hub so teams can review status and troubleshoot failures. The monitoring hub provides a centralized place to view recent activity, inspect run details, and understand whether jobs succeeded, failed, or are still running. Figure: Monitor scheduled User Data Function executions in Fabric Monitoring Hub. Conclusion Scheduled User Data Functions make it easier to automate recurring business processes directly within Fabric. Whether you're validating data, processing customer feedback, synchronizing systems, or generating operational alerts, scheduling helps keep business logic close to your data while simplifying operations. Learn more about Job scheduler in Microsoft Fabric and User data functions overview to get started.Bring your Azure Monitor and AWS Glue data to OneLake! (Preview)
Microsoft Fabric is expanding the reach of Microsoft OneLake with two new mirroring capabilities: Mirrored Azure Monitor and Mirrored AWS Glue catalog. These previews make it easier to bring operational telemetry and tables from external catalogs into Fabric while minimizing data movement and integration complexity. Bring more of your data estate into OneLake Organizations manage data across clouds and platforms. As two examples, operational telemetry may live in Azure Monitor, and some lakehouse environments may use AWS Glue Data Catalog to organize Apache Iceberg tables stored in Amazon S3. Bringing these systems together traditionally requires custom ingestion pipelines, duplicate storage, and ongoing infrastructure management. Fabric mirroring provides a simpler model: it connects to external systems and reflects data or metadata into OneLake and Fabric experiences, helping teams analyze information across platform boundaries. Mirror your Azure Monitor data into Fabric Mirrored Azure Monitor brings tables from Log Analytics workspaces into OneLake without duplicating the operational data. It connects OneLake to the Log Analytics storage used by Azure Monitor, allowing teams to combine telemetry with business data already available in OneLake. This creates opportunities for: Operational analytics across application and infrastructure signals. Cross-domain reporting that connects service health with business outcomes. Real-time intelligence scenarios using KQL. AI-powered reasoning across operational and business data. Because the data remains governed by Azure Monitor and is accessed without replication, teams can avoid building a second ingestion and storage path merely to analyze telemetry in Fabric. Bring Iceberg tables from AWS Glue into OneLake Mirrored AWS Glue catalog is designed for organizations that use AWS Glue Data Catalog for their Apache Iceberg tables backed by AWS S3. Users connect OneLake to AWS Glue, select supported Iceberg tables, and have their Iceberg tables automatically show up in OneLake for use in Fabric. The experience is intended to enable: Unified discovery of Glue-cataloged Iceberg data with other OneLake data. Faster onboarding of cataloged data into Fabric analytics experiences. Cross-cloud analysis using Fabric workloads such as Power BI, Data Warehousing, Data Engineering, and Data Science. The underlying data remains in its existing storage location, while Fabric uses mirrored metadata and OneLake shortcuts to make supported Iceberg tables available across the platform. Open by design These capabilities continue Microsoft’s commitment to open data architectures and cross-platform interoperability. The mirrored catalog feature establishes a metadata-based, zero-copy pattern for bringing external catalog-managed data into OneLake. Mirrored AWS Glue catalog and Mirrored Azure Monitor extends the mirrored catalog approach to additional providers and environments. Get started Refer to the following steps to try out mirroring for these new sources today! Mirrored Azure Monitor To set up mirroring for your Azure Monitor tables from your Log Analytics workspace: Create a new Mirrored Azure Monitor item in Fabric. Connect it to a supported Log Analytics workspace. Select the tables you want to make available. Start exploring and analyzing the mirrored tables using Fabric experiences. Check out the Mirror Azure Monitor in Microsoft Fabric (preview) documentation for more guidance. Mirrored AWS Glue catalog To set up mirroring for Iceberg tables from your AWS Glue catalog: Create a new Mirrored AWS Glue catalog item in Fabric. Connect to AWS Glue Data Catalog using a supported authentication method. Select the Iceberg tables you’d like to mirror into OneLake. Start exploring and analyzing the mirrored tables using Fabric experiences. Check out the Mirrored AWS Glue catalog (preview) documentation for more information. Mirrored Azure Monitor and Mirrored AWS Glue catalog help bring more of your data estate into OneLake, reducing the complexity of traditional integration projects and making it easier to work across operational, business, and cross-cloud data. We want your feedback! Try the previews today! Share your feedback through the Fabric Ideas site and Microsoft Fabric Community!Fabric Runtime Release Channels
Fabric Runtime Release Channels provide a structured and transparent way for customers to test upcoming runtime changes before they become the default. This feature helps organizations validate their production workloads early with these new changes in early access, avoid unexpected disruptions, and gain better control over Spark runtime upgrades. Instead of receiving silent updates that might break your production workloads, opt in to an early access release channel, test your workloads in a development or staging environment, and confirm compatibility before the update becomes default.Job concurrency and queue monitoring for Spark, now with a capacity-level view (Generally Available)
Spark jobs don't always start immediately. When jobs are queued or throttled, it can be difficult to determine if the cause is workspace limits, capacity saturation, or contention from other workspaces sharing the same capacity. Job concurrency and queue monitoring for data engineering are now generally available, and it includes a new capacity-level view that shows exactly how each workspace consumes your capacity over time.
