From factory floor to Fabric: Securely stream private data with Eventstream
Bring private operational data into Real-Time Intelligence Real-time operational data is often generated in places that are intentionally isolated from the public internet. Factory MQTT brokers, enterprise Kafka clusters, database change data capture sources, and messaging systems may live in an on-premises network, an Azure virtual network, or a private environment in another cloud. Microsoft Fabric Eventstream virtual network and on-premises support gives streaming connectors a managed path to approved private sources. The source stays behind its network boundary while Eventstream retrieves events for transformation, routing, analytics, and action. Network teams continue to control connectivity, and data teams use a shared network reference in Fabric instead of exposing a public endpoint. A manufacturing industry data scenario Consider a manufacturing environment where sensors, machines, and vehicles publish events every second to an MQTT broker in the factory network. Operations teams want to monitor equipment health, detect production issues, and respond to changing conditions. The analytics team wants to route those events through Eventstream to Eventhouse and Lakehouse, then use them in real-time dashboards and alerts. The MQTT broker is private by design. The security team doesn't permit inbound access from the public internet, and the broker has no publicly routable endpoint. The data team needs a connection model that reaches the broker through the organization's approved network path. A challenge shared across industries The challenge is common across industries. Valuable operational sources often live in on-premises networks, Azure virtual networks, or private environments in other clouds because those systems were intentionally designed not to be publicly accessible. Eventstream virtual network and on-premises support provide a secure, managed pathway for connecting those sources to Fabric. Network administrators retain control of the network boundary, while data engineers can use the approved streaming virtual network data gateway when creating Eventstream connections. Use an Azure virtual network as the controlled bridge Rather than making the MQTT broker public, data team partners with the networking team to use a customer-owned Azure virtual network as an intermediary bridge. This is the core of the Eventstream virtual network architecture. The team registers the Microsoft.MessagingConnectors resource provider, creates an Azure virtual network, and prepares a subnet delegated to Microsoft.MessagingConnectors. Because the MQTT broker is in the factory network, the networking team connects that network to Azure through the organization’s private connectivity approach, such as VPN or Azure ExpressRoute. Before moving forward, the team verifies that a client in the Azure virtual network can reach the MQTT broker. The broker remains private, but the Azure virtual network now has a secure route to it. Enable Fabric permission to use the network The private route now exists, but Eventstream still needs permission to use the Azure virtual network. Data team then works with Fabric admin to enable the workspace identity for the Fabric workspace that contains the Eventstream. The Azure administrator then grants the workspace identity the Network Contributor role on the Azure virtual network. Think of this as authorizing fabric to use approved network path. The network team created the bridge. The role assignment authorizes the Fabric workspace to use the prepared Azure VNet for connector injection. Bring the network reference into Fabric Inside Fabric, data team creates a streaming virtual network data gateway from Manage connections and gateways. The gateway represents the Azure virtual network and subnet resources within Fabric and passes the network information needed for streaming connector virtual network injection. The streaming virtual network data gateway is not a traditional gateway server. It is a Fabric reference to the Azure network resources already prepared by the networking team. d selects the prepared Azure VNet and delegated subnet. Connect and publish the MQTT source With the gateway available, data engineer returns to Eventstream and adds an MQTT source. During connection setup, select the streaming virtual network data gateway, supply the MQTT connection information and credentials, complete the source configuration, and publish the Eventstream. When the Eventstream is published, Fabric injects the streaming connector into the delegated subnet. The connector makes an outbound connection through the private route to retrieve MQTT events. No inbound public endpoint is required on the factory broker. Eventstream can then transform and route the events to Fabric destinations. Turning factory signals into real-time intelligence Once the Eventstream is running, factory telemetry starts flowing into Fabric. Data engineering team can transform and route the events to Eventhouse for real-time analytics, persist data in Lakehouse, create operational dashboards, and configure alerts for important conditions. The factory systems remain behind the company’s network boundary while the analytics team gains access to the operational signals needed for timely decisions. Plan for operational boundaries This connection model manages connector placement, but it doesn't replace the organization's network design. Teams still own private routing, DNS resolution, firewall rules, source credentials, certificates, and source availability. The same model applies to many supported streaming sources, including Apache Kafka, Azure Service Bus, Amazon Kinesis Data Streams, Google Cloud Pub/Sub, HTTP, and several CDC connectors. Connector-specific authentication and network requirements still apply. For prerequisites and the complete configuration sequence, follow the Eventstream streaming connector virtual network and on-premises support guide. Next steps Review the architecture with your networking and Fabric administrators, then identify one private streaming source for a controlled validation. Confirm network reachability before configuring Eventstream, document the delegated subnet and workspace permissions, and verify the complete event path after publishing. If you find this blog helpful, please give it a thumbs-up! Have ideas for what you'd like to see next? Drop us a comment or reach out to [email protected] — we'd love to hear what real-time scenarios you're building and what topics you'd like us to cover in future posts. To suggest improvements or additional scenarios, submit feedback through Fabric Ideas.From Business Events, Fabric Events, and Azure Events to Real-Time Hub
Welcome to the eighth and final post in our Business Events, Fabric Events, and Azure Events series for Microsoft Fabric. This series takes you from foundational event-driven concepts to practical implementation patterns that help teams turn meaningful business moments into trusted signals, decisions, and actions across Fabric.Designing scalable Business Events in Microsoft Fabric
Welcome to the seventh post in our Business Events, Fabric Events, and Azure Events series for Microsoft Fabric. This series takes you from foundational event-driven concepts to practical implementation patterns that help teams turn meaningful business moments into trusted signals, decisions, and actions across Fabric.Five Business Event scenarios and the pattern they share
Welcome to the sixth post in our Business Events, Fabric Events, and Azure Events series for Microsoft Fabric. This series takes you from foundational event-driven concepts to practical implementation patterns that help teams turn meaningful business moments into trusted signals, decisions, and actions across Fabric.Reacting to Business Events with Activator and Eventhouse
Welcome to the fifth post in our Business Events, Fabric Events, and Azure Events series for Microsoft Fabric. This series takes you from foundational event-driven concepts to practical implementation patterns that help teams turn meaningful business moments into trusted signals, decisions, and actions across Fabric.Publishing Business Events from Fabric workloads
Welcome to the fourth post in our Business Events, Fabric Events, and Azure Events series for Microsoft Fabric. This series takes you from foundational event-driven concepts to practical implementation patterns that help teams turn meaningful business moments into trusted signals, decisions, and actions across Fabric.Design a Business Event schema that consumers can act on
Welcome to the third post in our Business Events, Fabric Events, and Azure Events series for Microsoft Fabric. Whether you're new to event-driven architectures or building enterprise-grade event-driven solutions, this series takes you from foundational concepts to real-world implementation patterns across Microsoft Fabric.Choose the right event pillar in Microsoft Fabric: Business Events, Fabric Events, and Azure Events
Welcome to the second post in our Business Events, Fabric Events, and Azure Events series for Microsoft Fabric. Whether you're new to event-driven architectures or looking to build enterprise-grade event-driven solutions, this series will take you from foundational concepts to real-world implementation patterns across Microsoft Fabric.Build event-driven architectures in Fabric with Real-Time Hub
Learn the fundamentals of event-driven architecture in Microsoft Fabric. Explore Real-Time Hub, understand the three Fabric event pillars, and discover how Business Events enable workloads to react to meaningful business signals in real time.Fabric July 2026 Feature Summary
Welcome to the July 2026 Fabric update! This month brings new capabilities across the Fabric experience, from improved deployment and governance experiences to expanded Spark, Eventstream, and Real-Time Intelligence functionality. Whether you're building data pipelines, managing analytics workloads, or monitoring real-time operations, these updates help you work more efficiently and get more value from your data. _______________________________ Events and Announcements Get Fabric certified for FREE This is your chance to take the DP-600 (Fabric Analytics Engineer) or DP-700 (Fabric Data Engineer) certification exams for free. As part of Data Days, we have over 100 live sessions, more than 5 contests and challenges, and dozens of study groups and learning opportunities. And free Fabric exam vouchers. Available now through August 10, 2026. Request your voucher. Join us for FABCON and SQLCON in Barcelona, September 28 – October 1, 2026 Explore what’s possible with Microsoft Fabric and get up to speed on the latest in SQL, analytics, and AI. From 130 sessions and 4 keynotes to workshops, the expo, community spaces, and the Power BI DataViz World Championships, this is where the data community comes together. Learn directly from Microsoft and community experts shaping the future of Fabric and SQL. Register now and save €200 with code FABCMTY200. Fabric Platform Fabric-CICD tool v1.2.0 – new bulk publish mode (Preview) The June release of the fabric-cicd Python library, v1.2.0 introduces bulk publish mode. It lets fabric-cicd publish multiple items in a single API call using the Fabric bulk import API instead of publishing each item individually through a separate API call. This can make deployments more efficient. Why this matters Because dependencies are managed by the API during publication, bpublishinglish reduces the rigidity of item-type-based staging and better supports cross-item dependencies when logical ID references are used. It can reduce the number of item-specific parameter values you need to configure. For unsupported scenarios, fabric-cicd automatically falls back to the standard publishing flow, so you can try bulk publishing without manually switching deployment paths. To learn more, refer to fabric-cicd bulk option documentation and the v1.2.0 release notes. Change Git branch with at least the contributor role (Preview) Fabric Git integration now let’s any workspace member with at least the Contributor role switch the workspace's connected Git branch. Previously, this action required the workspace Admin role, which forced developers to either be over-permissioned or wait on an admin every time they moved between branches. Why this matters Developers no longer need Admin rights to switch branches. Removes a key blocker in the branch-out to existing workspace flow. Keeps developers as Contributors, aligned with least privilege governance. Fewer hand-offs to admins mean faster time to code. You can find this new capability under the Git integration settings. To learn more about Fabric Git integration new setting, refer to the Allow Contributors and Members to switch branches documentation. We refreshed the Fabric CI/CD documentation this month to make it easier to get started and to follow best practices: New CI/CD intro page — a reworked Introduction to CI/CD in Microsoft Fabric that walks through the platform layer by layer with a new enterprise reference architecture. New best practices guide — Understand the best practices for Fabric CI/CD brings together practical guidance for structuring workspaces, branching, and promoting content safely across environments. Auto-bind for Git integration — new guidance on cross-workspace dependency binding, covering how item dependencies automatically rebind when you branch out or update from Git, and which dependency types are supported. Actionable recommended actions for data owners in the OneLake catalog The Govern tab in the OneLake catalog gives data owners a health view of their data estate, along with recommended actions to improve it, such as increasing sensitivity label coverage, removing items that are no longer in use, handle failed refresh, and more. Until now, these recommendations have told data owners what to improve, but not which items were affected, leaving them to track the relevant entities down manually before they could act. That gap turned a clear recommendation into a manual investigation, and the action often stalled before it started. This release closes that gap. When you open a recommended action, you now see the specific items behind the recommendation, with everything you need to act on them in one place, and a direct path to open item details and resolve the issue at the source. Selecting a recommended action now opens a dedicated view that includes: A breakdown of the affected items — a visual summary of how many of your items are impacted. Why it matters — a short explanation of the governance impact. How to fix it — clear, step-by-step instructions for resolving the recommendation. Newly added A complete list of affected items — displayed in a table with key details, including Name, Last Refreshed, Owner, Location, Endorsement, and Sensitivity, making it easy to review and prioritize actions. Filter and search — narrow the list by keyword or filters to focus on the items you want to handle first. Open item details — jump straight to any affected item to make the change, instead of searching for it across your workspaces. The same enriched experience applies across the recommended actions in OneLake catalog Govern, helping you act on each one without leaving the catalog: Increase Sensitivity label coverage — find and label items that create potential security risks while unlabeled. Remove unused items — review items that weren't accessed or refreshed recently to keep your estate organized and reduce costs. Investigate items that failed to refresh — see which items failed to refresh so your data stays current and reliable. Add descriptions to your endorsed items — surface endorsed items without a description so people can better understand and use them. Apply relevant tags to your items — make items more discoverable by tagging the ones that are missing tags. Why it matters Governance only improves when recommendations turn into action. By bringing the affected items and a direct way to reach them into every recommended action, OneLake catalog Govern removes the guesswork from following through. Data owners can move from understanding a recommendation to resolving it in just a few steps, keeping their data estate secure, organized, discoverable, and trustworthy with far less effort. Learn more about the OneLake catalog Govern tab recommended actions. Data Engineering Microsoft Fabric Runtime 2.0 (Preview) Based on feedback received directly from our customers and partners, we have upgraded Fabric Runtime 2.0 to the latest and compatible stack: Apache Spark 4.1 Delta Lake 4.2 Python 3.13 These upgrades bring access to the newest enhancements, performance improvements, and ecosystem innovations, while also providing a longer support window for enterprise customers to build large-scale analytics and AI workloads on Fabric. With Runtime 2.0, customers can take advantage of: Latest Spark and Delta Lake capabilities. Improved compatibility across the modern data ecosystem. Better developer productivity and runtime performance. Future-ready platform investments aligned with long-term supportability. Learn more about Fabric Runtime 2.0. Fabric Runtime Release Channels (Preview) Fabric Runtime Release Channels provide a structured and transparent way for customers to test upcoming runtime changes before they become the default. This feature helps organizations validate their production workloads early with these new changes in early access, avoid unexpected disruptions, and gain better control over Spark runtime upgrades. Instead of receiving silent updates that might break your production workloads, you can opt in to an early access release channel, test your workloads in a development or staging environment, and confirm compatibility before the update becomes default. How release channels work Each Spark runtime has at least two public release channels: Default channel — This production-grade channel runs the default version of the runtime. All users automatically use this channel unless they opt in to early access. Early access channel — This production-grade channel includes upcoming updates and library changes that are scheduled to become the next default channel. You can opt in to test your workloads against upcoming changes. Once the designated validation window ends, the early access release channel automatically gets promoted to become the new default, and a fresh early access channel is introduced with another set of new changes — continuing the cycle. This model gives you a predictable testing window before changes become default for everyone. Why release channels matter Spark runtime updates can include library upgrades, security patches, dependency changes, or even operating system upgrades. While all updates pass internal quality checks before release, those checks can't capture all customer-specific variations and use cases. Early access channels let you identify potential issues early and work with Microsoft by creating a support ticket to address them before updates affect your production environment. Benefit Description ✔ Predictable updates Customers know exactly when a new runtime becomes available and have time to validate against it. ✔ Reduced risk Testing workloads on early access ensures compatibility before changes reach production. ✔ Better visibility Customers can easily tell which runtime version they're running, reference release notes, and verify upgrade timing. ✔ Improved quality and security You receive well-tested builds with security patches applied faster, giving you confidence in runtime stability. Learn more about Fabric Runtime Release Channels: Fabric Runtime Release Channels. Spark Diagnostic Emitter: Spark 4.1 Runtime Support and New Log Ingestion API The Fabric Apache Spark Diagnostic Emitter is now supported on the Fabric Runtime with Apache Spark 4.1. Customers can collect driver logs, executor logs, Spark event logs, and metrics from workloads running on the latest runtime and route them to Azure Event Hubs, Azure Blob Storage, or Azure Log Analytics — using the same spark.synapse.diagnostic.emitter.* configuration model, so existing emitter setups carry forward as customers upgrade. The emitter also now supports the Azure Monitor Log Ingestion API for sending diagnostics to Log Analytics, available on both Spark 3.5 and Spark 4.1 runtimes. The new AzureLogIngestion emitter type replaces the legacy HTTP Data Collector API path, providing a structured ingestion model with DCR/DCE-based authentication, schema definition, and routing into custom Log Analytics tables. Customers currently on the legacy AzureLogAnalytics type are encouraged to migrate — migration involves creating Data Collection Rule and Data Collection Endpoint resources and updating the Spark properties in the Fabric Environment. To learn more, refer to the Collect logs and metrics with Azure Log Analytics for migration guidance and Spark Diagnostic Emitter documentation. The Fabric Spark Operations Skill — AI-Assisted Spark Diagnostics, Open Source (Preview) The Fabric Spark Operations Skill (spark-operations-cli) is now available in the open-source Skills for Fabric library on GitHub (microsoft/skills-for-fabric). The skill brings AI-assisted, read-only diagnostics to Spark workloads in Fabric — troubleshoot failed notebooks and Spark jobs, stuck Livy sessions, and performance bottlenecks (OOM, shuffle, skew) in plain English from GitHub Copilot CLI, Claude Code, VS Code, Cursor, or other compatible AI tools. It returns severity-ranked findings with root cause analysis and fix recommendations, and includes an automated diagnostic workflow spanning job triage, log mining, Spark Advisor findings, and mitigations. Setup takes minutes: install the skill, run az login, and ask, "Why did my notebook fail last night?" Get started with Skills for Fabric in GitHub. Faster Python UDFs, Scala UDFs, and complex data types in the native execution engine (Generally Available) The native execution engine in Microsoft Fabric, which is generally available, now accelerates Python user-defined functions (UDFs), Scala UDFs, and complex data types such as arrays, maps, and structs. You get faster Spark processing for expressive code without changing your existing notebooks or jobs. Python UDFs have historically carried serialization overhead as data moves between the JVM and Python worker processes. The native execution engine optimizes that data path and keeps vectorized processing intact, so scalar and Pandas (vectorized) UDFs run faster automatically. Scala UDFs and queries that work with nested, complex data types benefit from the same native acceleration. Why this matters Faster UDF execution with no code changes. Vectorized Pandas UDFs see the largest gains. Complex types (arrays, maps, structs) run natively. Existing notebooks and jobs benefit automatically. To use Efficient Scaledown, enable the native execution engine on your Fabric Spark pool or environment. Existing Python and Scala UDFs and queries over complex types are accelerated without any code changes. To learn more, refer to the Python UDFs, Scala UDFs, and complex data types in the native execution engine documentation. Efficient Scaledown with the remote shuffle manager (Preview) Efficient Scaledown decouples Spark shuffle data from executor lifetime in Microsoft Fabric. Instead of pinning shuffle output to local executor disks, Fabric Spark routes large shuffles to Azure Blob Storage and migrates blocks off executors before they're released. Clusters scale down faster, compute costs drop, and jobs become more resilient — with no changes to your queries, notebooks, or pipelines. The feature combines four cooperating capabilities: the Remote Shuffle Manager writes and reads shuffle data to Azure Blob Storage; Shuffle Migration moves blocks off an executor before decommissioning instead of dropping them; a Decision Layer routes small shuffles to local disk and large shuffles to remote storage per stage; and AQE Shuffle Write lets Adaptive Query Execution shape partitioning the first time. Why this matters Clusters scale down faster after demand drops. Lower compute cost from quicker executor release. More resilient jobs with fewer stage retries. No changes to queries, notebooks, or pipelines. To use it, enable the native execution engine and run on Runtime 1.3 (Apache Spark 3.5) or later; autoscale is recommended. Remote Shuffle Manager spark.conf.set("spark.remote.shuffle.enabled", "true") Decision Layer — per-stage routing of local vs. remote shuffle spark.conf.set("spark.sql.rsm.decisionlayer.enabled.level", "stage") AQE participates in shuffle write spark.conf.set("spark.sql.adaptive.shuffleWrite.enabled", "true") Shuffle Migration on executor decommission spark.conf.set("spark.storage.decommission.shuffleBlocks.enabled", "true") spark.conf.set("spark.storage.decommission.shuffleBlocks.cleanup", "true") spark.conf.set("spark.storage.decommission.shuffleBlocks.migrateToFallbackStorage", "true") spark.conf.set("spark.storage.decommission.fallbackStorage.cleanUp", "true") To learn more, refer to the Efficient Scaledown and remote shuffle manager in Microsoft Fabric documentation. Customer-managed key encryption for Spark shuffle data on disk (Generally Available) Microsoft Fabric Spark now generally supports customer-managed keys for Spark jobs through a disk encryption set. Shuffle data written to cluster disks during a Spark job is encrypted with a key you supply and control, giving you ownership of the encryption material that protects intermediate data at rest. With a disk encryption set configured, the cluster disks that hold Spark shuffle data are encrypted using your customer-managed key rather than a platform-managed key. You manage the key lifecycle — including rotation and access — in your own key vault, so encryption of intermediate Spark data follows your organization's key management policies. Why this matters You control the key protecting shuffle data. Intermediate Spark data on disk is encrypted at rest. Key lifecycle and rotation stay in your control. Encryption aligns with your key management policies. To use it, configure a disk encryption set backed by your customer-managed key and associate it with your Fabric Spark configuration. Spark jobs then encrypt shuffle data on cluster disks with your key. To learn more, refer to the Customer-managed key encryption for Fabric Spark documentation. Query your data instantly with the Lakehouse Query Explorer (Generally Available) The Lakehouse Query Explorer is a new, fully integrated query editor built directly into the Lakehouse experience in Microsoft Fabric. You can now write and run Spark SQL queries right where your data lives — no need to switch to a SQL endpoint or spin up a notebook for quick exploration. Whether you’re validating datasets, iterating logic, or exploring patterns, Query Explorer keeps you in flow. orer with the new integrated Query Explorer. Key capabilities Fast, lightweight Spark execution powered by the Lakehouse Livy endpoint. IntelliSense + rich editing experience for faster query authoring. Query across schemas and lakehouses in a single tab. Built-in results grid + inline charts to explore results instantly. Multiple dynamic tabs to analyze different slices of data side-by-side. From quick lookups to multi-table exploration, Query Explorer makes working with Lakehouse data faster and more intuitive—right from the explorer. Learn more and get started with the Lakehouse Query Explorer documentation. Analytics and Insights for Materialized lake views (Generally Available) Analytics and Insights bring continuous refresh intelligence to your lakehouse — two new tabs beside Recent run(s) that shift you from reacting to individual failures to staying ahead of performance drift, rising costs, and silent inefficiencies across your entire materialized lake view estate. The Recent run(s) page tells you what happened in a single execution, including which views succeeded, which failed, and how long the job took. That's valuable when something breaks, but it doesn't answer the questions that matter most for day-to-day operations. Are my durations stable or slowly climbing? Is a new error class appearing and spreading across views? Are my schedules still aligned with how often upstream data actually lands? Am I paying for full refreshes on views that could run incrementally? These are the questions that separate a well-tuned deployment from one that quietly accumulates cost and risk until something finally breaks loudly enough to notice. The Recent run(s) page answers all of them. The Analytics tab transforms your run history into trend lines, distributions, and comparisons you can read at a glance: duration trajectories, success-rate shifts, error-class frequency over time. The Insights tab goes further by watching those same patterns the way a seasoned reliability engineer would, recognizing the signatures behind slow, failing, or wasteful runs, and providing a prioritized list of worthwhile changes. Each recommendation names the affected view, explains why it's flagged, and estimates the payoff in runtime savings, so you can act in seconds rather than investigate for hours. Together, they move refresh management from a reactive, break-fix posture to a proactive, continuously improving one — keeping your materialized lake views fast, healthy, and cost-efficient as your estate grows. To learn more, refer to analytics charts, insight categories in Materialized Lake Views. Introducing Event-Driven Refresh for Materialized Lake Views (Preview) Event-driven refresh brings responsive refresh intelligence to your lakehouse — a new scheduling mode alongside time-based schedules that shifts you from refreshing on the clock to refreshing the moment your data is ready, so your materialized lake views reflect reality instead of an arbitrary calendar. Time-based schedules tell your views when to run: every hour, every morning, every night. That's dependable when upstream data lands like clockwork, but it doesn't answer the questions that matter most for day-to-day operations. Did the ingestion pipeline that feeds this view finish before I refreshed it? Am I recomputing gold-layer views on a fixed cadence while the source data only changes twice a day? Am I paying for refreshes that run before new data has even arrived — or worse, serving stale results because the next scheduled slot is still hours away? These are the questions that separate a refresh strategy tuned to your data from one that quietly burns compute on empty runs and lags the moments that matter. Event-driven refresh answers all of them. You bind a view or a lineage sub-chain to the events that should drive it, and we support two types of events: OneLake events — all file and folder events are supported, so a refresh can fire the moment data lands in OneLake (file or folder creation, update, and more). Job events — Pipeline and Notebook events are supported, so a refresh can fire when the ingestion job that feeds your views completes. The moment an event fires, Fabric resolves the dependency chain and refreshes exactly the views that depend on it. Each trigger names the source event, scopes precisely to the affected lineage, and can react to success or failure, so a stalled upstream job never silently cascades into stale downstream reports. Together with multi-schedule support, event-driven refresh moves refresh management from a fixed-clock, guess-the-cadence posture to a responsive, data-driven one — keeping your materialized lake views fresh the instant new data arrives, and idle when it hasn't, as your estate grows. Learn more and get started with the Schedule a Materialized Lake View Refresh documentation. Data Science AI functions: new models, no package dependency, better usage stats (Generally Available) Fabric AI Functions now use gpt-5-mini as the default model, with “low” reasoning enabled. This powers AI Functions across pandas, PySpark, Data Warehouse, and Dataflows Gen2. For more sophisticated transformations, users may configure gpt-5.1 or tune the reasoning_effort parameter for additional compute and higher-quality results. The gpt-4.1 model has been retired. Pipelines pinned to gpt-4.1 have migrated to gpt-5.1, and those pinned to gpt-4.1-mini migrated to gpt-5-mini. We’ve also simplified PySpark AI Function chaining. The PySpark .ai interface now stays bound to the result schema, so chains like summarize → classify no longer require intermediate DataFrames. In addition, PySpark now supports df.ai.stats for detailed token usage after any AI function call, including reasoning token breakdowns. For pandas, AI Functions no longer require the openai-python package. Capacity-limited rows are surfaced as CapacityExceededResult, enabling clean retries via aifunc.split_results. To learn more, refer to the AI Functions documentation. Data Warehouse Lakehouse table health check (Generally Available) Lakehouse Table Health Check gives you a simple, T-SQL–based way to validate whether your Lakehouse tables are optimized for the SQL analytics endpoint. A single stored procedure surfaces common layout issues, such as small files and fragmentation, offering the insights you need to determine if your tables need to be optimized. You can integrate health checks into pipelines and operational workflows, enabling proactive, at-scale optimization instead of reactive troubleshooting. -- Run a health check on a Lakehouse table from the SQL analytics endpoint EXEC sp_get_table_health_metrics 'dbo.FactSales'; aluate table health. To learn more about sp_get_table_health_metrics and how to integrate it into your Pipelines to optimize your table only if anomalies are detected, refer to sys.sp_get_table_health_metrics (Transact-SQL). Scalar User‑Defined Functions - Procedural computation for analytical SQL (Preview) Scalar User-Defined Functions now support procedural computation — including loops, multiple return paths, and rich IF/THEN/ELSE branching — running natively within the warehouse engine. Computation-based Scalar UDFs are designed for analytical query shapes, integrating naturally with CTEs, GROUP BY, HAVING, and ORDER BY, and executing efficiently at data warehouse scale. Defining business rules once, in SQL, and reuse them across queries, reports, and pipelines. To learn more, refer to the Create Function documentation. Usage-based resource estimations (Generally Available) The Query Optimizer uses learned resource estimation to correct underestimation in T-SQL query plans. It saves actual cardinalities from past executions and automatically adjusts row count estimates in subsequent runs. With this release, the optimizer will also begin correcting overestimations — closing the loop in accurate cardinalities, leading to more efficient resource requests and improved concurrency. Real-Time Intelligence Improved tile error experience in Real-Time Dashboard (Generally Available) We improved the way tile errors appear in Real-Time Dashboards to make them clearer, calmer, and easier to act on. Instead of showing a disruptive red error state, tiles now use a neutral grey error state with a short category header, such as Syntax error, Semantic error, Data source issue, Network error, or Something went wrong. Users can select Details directly from the tile to open a popover with the full engine error message, shown exactly as received. This keeps the dashboard readable while still making the technical details easy to access, copy, and share when needed. This update helps users understand what went wrong faster, while ensuring that one failed tile does not block or visually overwhelm the rest of the dashboard. To learn more, refer to the Troubleshoot Real-Time Dashboard Tile Errors documentation. Folders in Eventhouse tree (Generally Available) Folder support in the Eventhouse tree is now available through the UI. Previously, folders could only be created and managed via code. With this update, you can now organize your Eventhouse directly from the tree, making it easier to manage assets at scale. You can group tables, shortcuts, materialized views, functions, and data streams into a structured hierarchy, improving navigation and reducing clutter. Key capabilities Create, rename, and delete folders from the UI. Organize assets into folders. Move items via the context menu (⋯ → Move to). Create folders inline while moving items. This allows for a simplified and more intuitive way of managing growing Eventhouses. To learn more, refer to the Manage and monitor a KQL database table documentation. Eventstream connector private network support (Generally Available) Data is a critical asset for organizations, and access to real-time data is increasingly essential. However, many high-value data sources reside in private network environments — cloud virtual networks or on-premises infrastructure — particularly in highly regulated industries such as banking, finance, and telecommunications, where strict security and compliance requirements are mandatory. Eventstream's private network support establishes a secure, managed bridge using your Azure virtual network and VNet injection, allowing Eventstream streaming connectors to run inside your virtual network and reach private sources without opening them to the public internet. Whether your data sources reside in on-premises networks, private networks on third-party cloud services, or private networks on Azure, you can connect the bridge Azure virtual network to your source's private network using suitable connectivity options — such as VPN or ExpressRoute for on-premises environments, and private endpoints or network peering for Azure-based sources — enabling Eventstream connectors to securely ingest real-time data from these protected environments into Fabric. The solution leverages a new concept — Streaming virtual network data gateways — which abstracts the bridge Azure virtual network and subnet resource within Fabric. By creating a connector with a streaming virtual network data gateway associating it with your connection, the Eventstream connector is provisioned within your virtual network, ensuring secure communication with your private data sources. Once real-time data from your private network source is securely brought into Fabric Eventstream, you can fully leverage the comprehensive analytics tools in Fabric Real-Time Intelligence to power your real-time scenarios with enterprise-grade security. To learn more about configuration and advanced scenarios, refer to the Eventstream private network streaming guide. Azure Event Hubs source in Eventstream now supports workspace identity authentication (Preview) Currently, when configuring an Azure Event Hubs source in Eventstream, the only supported authentication method is Shared Access Key — a connection string containing static credentials. While simple to set up, shared access keys present several security risks in production environments: they have unlimited lifetime unless manually rotated, provide no per-user or per-application identity, and if accidentally leaked through source code, configuration files, or third-party sharing, grant full access to anyone who obtains them. Revoking a compromised key requires regeneration, which disrupts all services depending on it — with no straightforward way to audit which clients used the key. To address these challenges, we're introducing Workspace Identity as a new authentication option for the Azure Event Hubs source connector (Extended features) in Eventstream (Preview). A Fabric workspace identity is an automatically managed service principal associated with your workspace. Fabric manages the credentials entirely — there are no secrets to store, rotate, or risk leaking. It integrates with Microsoft Entra ID, providing identity-based access with full audit trails, fine-grained role-based access control, and automatic credential lifecycle management. To use workspace identity authentication with your Azure Event Hubs source in Eventstream: Navigate to your workspace settings and create a workspace identity on the Workspace identity tab. In your Azure Event Hub namespace, assign the appropriate role (e.g., Azure Event Hubs Data Receiver) to the workspace identity's service principal. When adding an Azure Event Hubs source in Eventstream, select Workspace Identity as the authentication method — no connection string or key is needed. Eventstream will automatically obtain tokens using the workspace identity to securely connect to your Event Hub, eliminating credential management overhead while strengthening your security posture. To learn more about the configuration, refer to the Azure Event Hubs source extended connector configuration documentation. Custom CA and mTLS support in Eventstream streaming connectors (Generally Available) Fabric Eventstream under Real-Time Intelligence provides various streaming connectors, enabling the integration of real-time data from popular sources into Fabric. When the Eventstream connector client establishes a connection with sources, it is required to implement TLS or mTLS encryption to fulfil the necessary security standards. Many organizations use certificates issued by private or internal Certificate Authorities or require mutual TLS (mTLS) authentication where both the client and server verify each other's identity before transmitting data. Without custom CA and mTLS support, Eventstream connectors cannot connect to these secured source systems. The Custom CA and mTLS support feature, is now generally available for MQTT, Apache Kafka, AWS MSK, and Confluent Cloud for Apache Kafka source connectors. Customers can specify their custom CA and client certificates managed in their own Azure Key Vault when configuring their source in Eventstream. Once specified, Eventstream connector will fetch the certificates from the customer's Azure Key Vault and use them to establish a mutually authenticated, encrypted connection — enabling secure, compliant real-time data ingestion across all supported streaming sources. To learn more about the configuration, refer to the Eventstream sources overview page and choose the corresponding source. Introducing the Oracle CDC connector for Eventstream (Preview) Eventstream now introduces the Oracle Database Change Data Capture (CDC) connector, enabling you to stream database change events directly from any Oracle Database — whether running in the cloud or on-premises — into Eventstream for real-time processing and analytics. Many organizations run important operational workloads on Oracle Database and need to react to changes as they happen. With the Oracle CDC connector, you can continuously capture change events from Oracle Database and bring them into Fabric without building custom polling applications or managing separate integration services. With the Oracle CDC connector, you can: Capture and stream databases changes from Oracle Database into Fabric in real time. Connect to Oracle databases running either on-premises or in the cloud. Process incoming change events using Eventstream transformations. Route processed change events to supported destinations such as Eventhouse, Lakehouse, Activator, or custom endpoints. This capability helps you build real-time analytics and event-driven applications from Oracle data. For example, you can route transaction changes to an Eventhouse Kusto table for operational analysis, send selected events to Activator for alerting, or combine Oracle change events with other streaming sources in the same eventstream. To learn more about Eventstream Oracle CDC connector, refer to Add Oracle Database CDC source to an eventstream (preview). Eventhouse update policies now support referencing accelerated shortcuts in update policies (Generally Available) Eventhouse update policies now support accelerated shortcuts in update policy queries, enabling ingestion-time enrichment scenarios. Use this for dimension lookups, such as enriching ingested fact events with customer, device, or product attributes stored in OneLake shortcut data. The shortcut-backed external table must have Query Acceleration Policy enabled, and Hot must cover all data. For update policy scenarios, set: .alter external table DimCustomer policy query_acceleration '{"IsEnabled":true,"Hot":"36500.00:00:00"}' Then join to it from the update policy query: .alter table EnrichedEvents policy update '[{ "IsEnabled":true, "Source":"RawEvents", "Query":"RawEvents | lookup kind=leftouter (external_table(''DimCustomer'')) on CustomerId","IsTransactional":true, "PropagateIngestionProperties":false }]' Processing uses the authorization context captured in the system-populated OwnerPrincipalDetails property: the user who creates or alters the update policy must have access to the shortcut data. This enables ingestion-time enrichment with governed shortcut data without separate orchestration. Shortcut Tables in Eventhouse Now Automatically Synchronize Schema Changes To help maintain consistency between source data and shortcut tables, the default schema synchronization behavior for Eventhouse shortcut tables is changing. Previously, schema changes made to the source table were not automatically propagated to the shortcut table unless schema synchronization was explicitly enabled. As a result, source and shortcut schemas could diverge over time. With this update, all new and existing shortcut tables in Eventhouse automatically synchronize schema changes from their source table by default, including: Adding new columns Changing column data types Renaming columns Deleting columns Automatic schema synchronization helps ensure that shortcut tables remain aligned with the source schema, preserving the latest business context and reducing manual maintenance. Customers who prefer to disable automatic schema synchronization can continue to control this behavior using the existing KQL management command: .create-or-alter external table ExternalTable kind=delta ( h@'https://storageaccount.blob.core.windows.net/container1;secretKey' ) with (AutoUpdateSchema=false) Note: Because schema changes are now automatically propagated, queries, dashboards, and downstream workloads may require updates if they reference columns that are renamed, removed, or otherwise modified in the source table. Investigator Insights in Operations Agent (Preview) When an anomaly is detected, understanding what caused it is often the hardest part. Investigator insights are designed to make that easier by analyzing the surrounding data and surfacing relevant context. When the operations agent detects that a rule has been met, there is an option to run an investigation in the background to identify correlated signals and patterns. Instead of manually digging through telemetry, you get a guided view into what changed, what stood out, and what may have contributed to the issue. This helps you move more quickly from detection to understanding. You can access these insights directly from Teams. Open the agent’s message and select Investigate further to generate a detailed analysis. The investigation provides a structured view of what happened: Investigation scope shows which tables were analyzed, whether from a single source or across related datasets. Key observations highlight the most important findings, including actual values, deviations from baseline, and notable trends or outliers. Pattern analysis surfaces meaningful changes around the time of the anomaly, such as dimensions with significant shifts, and clearly call out when no strong patterns are identified. Together, these insights help you quickly understand not just that something went wrong, but why it happened. To learn more, refer to the Operations Agent Actions documentation. Anomaly Detector Configurations Pane As you build on top of your data, it is often just as important to understand what already exists as it is to create something new. This update makes it easier to discover and build existing anomaly detection configurations. You can now view all anomaly detection configurations that have already been created for a given data source in one place. This lightweight experience gives you quick visibility into how anomaly detection is currently set up, helping you avoid duplicate work and better understand how others are using the data. From this view, you can explore existing configurations or create a new one if your use case is not yet covered. This makes it simple to extend existing setups or start fresh when needed, all without leaving the context of your data source. By making configurations easier to discover and reuse, this experience helps streamline workflows and ensures you can move quickly from exploration to action. To learn more, refer to the Anomaly Detection in Real-Time Intelligence documentation. Ingestion time stamp in Anomaly Detector (Preview) Working with anomaly detection often assumes your data already includes a clean, reliable timestamp. With this update, you can now use the system-generated ingestion time as the timestamp for anomaly detection. This means that even if your dataset does not include a dedicated timestamp column, you can still run analysis without needing to modify or preprocess your data. This is especially helpful for scenarios where events are ingested in real time or where timestamps are missing, inconsistent, or not trustworthy. Instead of blocking data preparation, you can rely on ingestion time to move forward with detection and start identifying meaningful patterns right away. By reducing setup requirements and removing a common dependency on source data quality, this capability makes it easier to apply anomaly detection across a wider range of use cases. To learn more, refer to the Anomaly Detection in Real-Time Intelligence documentation. Configuring Anomaly Detection without a Group by Column (Preview) With this update, Anomaly Detection now supports scenarios where you choose not to use a group by column. This provides an additional configuration option for datasets that already represent a single stream of data, allowing you to apply anomaly detection directly to the metric without first identifying a grouping dimension. This added flexibility helps the configuration experience better align with how your data is structured. Whether you are monitoring a single device, tracking a specific service, or analyzing a focused dataset, you can now create anomaly detectors without requiring a group-by column. At the same time, grouping remains available for scenarios where you want to monitor and compare multiple entities within the same dataset. To learn more, refer to the Anomaly Detection in Real-Time Intelligence documentation. Introducing Fabric Maps Tilesets: High-Performance Visualization for Large Geospatial Datasets (Generally Available) Have location-based data sitting in OneLake but no simple way to see it come alive on a map? With Fabric Maps Tilesets, you can now turn that data into fast, interactive map experiences directly inside Microsoft Fabric. The Tileset Builder lets you create map-ready PMTiles from OneLake data without custom code, manual exports, or separate geospatial infrastructure making it easier for teams to explore large geospatial datasets, uncover patterns, and bring location intelligence into the analytics workflows they already use. Organizations can rely on Microsoft Fabric to manage operational data, analytics workflows, and business reporting. When that data includes locations, routes, assets, boundaries, or events with a location context, teams need a simple way to visualize it on a map. Traditionally, this required separate geospatial pipelines, custom polling services, or manual exports. Leveraging a Fabric Map removes complexity by allowing organizations to create and refresh map-ready tilesets from data already stored in OneLake. Try creating your own Tileset using the following steps: Connect to a lakehouse and select source files Configure tileset metadata Configure layer settings Tileset schedule (Preview) Review and create tileset What is a Tileset? Tilesets are map-optimized representations of geospatial data. Instead of trying to load a large geospatial data file all at once, data is divided into small tiles that are loaded and rendered as needed while users zoom and pan across the map. This makes tilesets especially useful for large datasets, such as infrastructure networks, delivery routes, asset locations, service areas, or operational events. Fabric Maps Tileset Builder Capabilities Build map-ready tilesets directly from OneLake data. Visualize large geospatial datasets with high performance. Enable smooth, interactive map experiences at scale. Keep map content synchronized with source data. Eliminate manual exports and external geospatial processing workflows. Integrate native geospatial visualization into existing Fabric data workflows. Common scenarios Tilesets help teams turn large location-based datasets into fast, interactive map experiences. Utility and energy companies can visualize nationwide power line grid systems as a single dataset to monitor field assets and service coverage. Supply chain and logistics teams can explore daily routes, regional boundaries, and operational areas. Retailers can analyze store territories, expansion opportunities, and new developments. Because the data stays connected to Fabric, these map experiences become part of the broader analytics workflow — not a separate geospatial process. To get started, refer to How to create tilesets. Cross-domain intelligence with Azure Monitor data in Microsoft Fabric (Preview) Have you ever detected an issue in your systems but struggled to understand what it meant for the business? As systems grow more complex, this gap becomes harder to bridge. Incidents no longer affect just systems; they affect customers, revenue, and operations in real time. Azure Monitor Logs mirroring into Microsoft Fabric helps close that gap. In just a few steps, telemetry from Log Analytics workspaces becomes available in OneLake alongside business and operational data — without duplication and with near real-time availability. This creates the foundation for Cross-domain insights and actions: Bring observability, operational, and business data together in Eventhouse for real-time analysis. The same unified data can be used by Real-Time Dashboards for investigation and by operations agents to recommend and drive actions informed by both business and observability context. For example, an operations team can identify that a check-in kiosk outage is impacting high-value customers and act before customer impact grows. Advanced Fabric analytics Apply tools like Spark and Power BI for long-term analysis, machine learning, and a wide range of analytical scenarios. For example, an operations team can create a Power BI report showing trends in customer impact and cost, helping management make informed decisions on resource allocation and understand the true cost of application failures. Together, these capabilities help organizations move from isolated technical signals to business-aware insights, decisions, and actions. To learn more, refer to Cross-domain intelligence with Azure Monitor data in Microsoft Fabric (Preview). Until next month That's a wrap for the July 2026 Microsoft Fabric Monthly Update. As always, we'll continue sharing new capabilities, enhancements, and improvements across Microsoft Fabric in future monthly updates. Thank you for being part of the Fabric community!
