Get data from Apache Kafka topics via private networking
Current prerequisites limit to public Kafka instances which is not feasible for most enterprise customers. When will this limitation be lifted so that it can be used in private network setups, for example via gateway? https://learn.microsoft.com/en-gb/fabric/real-time-intelligence/event-streams/add-source-apache-kafka#prerequisites
NL to KQL/GQL generation
Hi Team, This is coming from Using Data Agents for NL to KQL/GQL generation | Microsoft Fabric Community. It would be great if we could have a feature where the data agent or the ontology agent can return just the grounded KQL/GQL query (or query variants) required to answer a NL question rather than actually execute it and summarize the result. This is important in use-cases like ours where retrieval needs to be split from query generation. Currently Fabric IQ doesn't provide a way to do that, and the only way seems to be to inspect the data agent's run steps which seems roundabout for something that might be asy to support natively. Please consider this feature request. Thanks!
From factory floor to Fabric: Securely stream private data with Eventstream
Bring private operational data into Real-Time Intelligence Real-time operational data is often generated in places that are intentionally isolated from the public internet. Factory MQTT brokers, enterprise Kafka clusters, database change data capture sources, and messaging systems may live in an on-premises network, an Azure virtual network, or a private environment in another cloud. Microsoft Fabric Eventstream virtual network and on-premises support gives streaming connectors a managed path to approved private sources. The source stays behind its network boundary while Eventstream retrieves events for transformation, routing, analytics, and action. Network teams continue to control connectivity, and data teams use a shared network reference in Fabric instead of exposing a public endpoint. A manufacturing industry data scenario Consider a manufacturing environment where sensors, machines, and vehicles publish events every second to an MQTT broker in the factory network. Operations teams want to monitor equipment health, detect production issues, and respond to changing conditions. The analytics team wants to route those events through Eventstream to Eventhouse and Lakehouse, then use them in real-time dashboards and alerts. The MQTT broker is private by design. The security team doesn't permit inbound access from the public internet, and the broker has no publicly routable endpoint. The data team needs a connection model that reaches the broker through the organization's approved network path. A challenge shared across industries The challenge is common across industries. Valuable operational sources often live in on-premises networks, Azure virtual networks, or private environments in other clouds because those systems were intentionally designed not to be publicly accessible. Eventstream virtual network and on-premises support provide a secure, managed pathway for connecting those sources to Fabric. Network administrators retain control of the network boundary, while data engineers can use the approved streaming virtual network data gateway when creating Eventstream connections. Use an Azure virtual network as the controlled bridge Rather than making the MQTT broker public, data team partners with the networking team to use a customer-owned Azure virtual network as an intermediary bridge. This is the core of the Eventstream virtual network architecture. The team registers the Microsoft.MessagingConnectors resource provider, creates an Azure virtual network, and prepares a subnet delegated to Microsoft.MessagingConnectors. Because the MQTT broker is in the factory network, the networking team connects that network to Azure through the organization’s private connectivity approach, such as VPN or Azure ExpressRoute. Before moving forward, the team verifies that a client in the Azure virtual network can reach the MQTT broker. The broker remains private, but the Azure virtual network now has a secure route to it. Enable Fabric permission to use the network The private route now exists, but Eventstream still needs permission to use the Azure virtual network. Data team then works with Fabric admin to enable the workspace identity for the Fabric workspace that contains the Eventstream. The Azure administrator then grants the workspace identity the Network Contributor role on the Azure virtual network. Think of this as authorizing fabric to use approved network path. The network team created the bridge. The role assignment authorizes the Fabric workspace to use the prepared Azure VNet for connector injection. Bring the network reference into Fabric Inside Fabric, data team creates a streaming virtual network data gateway from Manage connections and gateways. The gateway represents the Azure virtual network and subnet resources within Fabric and passes the network information needed for streaming connector virtual network injection. The streaming virtual network data gateway is not a traditional gateway server. It is a Fabric reference to the Azure network resources already prepared by the networking team. d selects the prepared Azure VNet and delegated subnet. Connect and publish the MQTT source With the gateway available, data engineer returns to Eventstream and adds an MQTT source. During connection setup, select the streaming virtual network data gateway, supply the MQTT connection information and credentials, complete the source configuration, and publish the Eventstream. When the Eventstream is published, Fabric injects the streaming connector into the delegated subnet. The connector makes an outbound connection through the private route to retrieve MQTT events. No inbound public endpoint is required on the factory broker. Eventstream can then transform and route the events to Fabric destinations. Turning factory signals into real-time intelligence Once the Eventstream is running, factory telemetry starts flowing into Fabric. Data engineering team can transform and route the events to Eventhouse for real-time analytics, persist data in Lakehouse, create operational dashboards, and configure alerts for important conditions. The factory systems remain behind the company’s network boundary while the analytics team gains access to the operational signals needed for timely decisions. Plan for operational boundaries This connection model manages connector placement, but it doesn't replace the organization's network design. Teams still own private routing, DNS resolution, firewall rules, source credentials, certificates, and source availability. The same model applies to many supported streaming sources, including Apache Kafka, Azure Service Bus, Amazon Kinesis Data Streams, Google Cloud Pub/Sub, HTTP, and several CDC connectors. Connector-specific authentication and network requirements still apply. For prerequisites and the complete configuration sequence, follow the Eventstream streaming connector virtual network and on-premises support guide. Next steps Review the architecture with your networking and Fabric administrators, then identify one private streaming source for a controlled validation. Confirm network reachability before configuring Eventstream, document the delegated subnet and workspace permissions, and verify the complete event path after publishing. If you find this blog helpful, please give it a thumbs-up! Have ideas for what you'd like to see next? Drop us a comment or reach out to [email protected] — we'd love to hear what real-time scenarios you're building and what topics you'd like us to cover in future posts. To suggest improvements or additional scenarios, submit feedback through Fabric Ideas.Support Creator-Independent Managed Identity and Ownership Transfer for Fabric Operations Agents
Microsoft Fabric Operations Agents should support an enterprise identity and ownership model that is independent of the individual user who originally created the agent. Today, an Operations Agent receives its own Microsoft Entra Agent ID, but the agent operates using the delegated identity and permissions of its creator. This creates an operational dependency on an individual employee account. For production environments, an Operations Agent may run continuously for months or years. The agent should not stop working or unexpectedly lose access because the original creator changes roles, loses access to a workspace, leaves the organization, or has permissions modified. Fabric should support: Creator-independent agent identity Managed identity or service-principal-style execution Explicit permissions assigned directly to the agent Transfer of agent ownership to another user or team Multiple agent owners or administrators Separation between agent owner, operator, approver, and notification recipient Ability to rotate or replace the execution identity without recreating the agent Clear visibility into which identity is used for every query and action Permission validation before an agent is started Alerts when the agent's required permissions are removed Audit history for identity and ownership changes Support for deployment across development, test, and production using environment-specific identities For example, a manufacturing Operations Agent may continuously monitor equipment telemetry and invoke a Fabric notebook or Power Automate workflow when abnormal conditions are detected. If the employee who originally created that agent leaves the project, the production agent should continue operating using an enterprise-managed identity rather than depending on the former employee's delegated permissions. Administrators should also be able to transfer ownership without recreating the agent, regenerating its playbook, or breaking its rules and downstream actions. This capability would make Operations Agents significantly safer for long-running enterprise automation and would align agent lifecycle management with established enterprise security, least-privilege, and service-identity practices.Add Schema Compatibility Policies and Breaking-Change Detection to Fabric Event Schema Registry
Fabric Event Schema Registry should support configurable schema compatibility policies so teams can safely evolve event contracts without breaking downstream Eventstreams, Eventhouse ingestion, applications, or other consumers. Today, schemas can be versioned, but compatibility enforcement is not yet available. This means producers can introduce changes that may unexpectedly break existing event-processing flows. Fabric should allow teams to define compatibility rules at the schema-set or individual-schema level. Requested capabilities should include: Backward compatibility Forward compatibility Full compatibility No-compatibility mode for intentionally breaking changes Automatic validation before a new schema version is published Clear identification of incompatible field additions, removals, type changes, and structural changes Breaking-change warnings Dependency and impact analysis showing which Eventstreams, Eventhouse tables, destinations, and applications use the affected schema Side-by-side schema diff between versions CI/CD validation API so compatibility checks can run before deployment Approval workflow for intentionally breaking changes Ability to mark older schema versions as deprecated with an effective retirement date For example, if a producer changes a field from integer to string, removes a required property, or changes a nested structure, Fabric should immediately show whether the new version is compatible with existing consumers before the schema is activated. The experience should also show exactly which downstream artifacts may be affected by the change. This would turn Event Schema Registry into a true enterprise contract-management capability rather than simply a centralized place to store schema versions. It would significantly reduce production failures caused by unexpected schema evolution and make Fabric Eventstream safer for large-scale real-time and event-driven architectures.Add Schema Compatibility Policies and Breaking-Change Detection to Fabric Event Schema Registry
Fabric Event Schema Registry should support configurable schema compatibility policies so teams can safely evolve event contracts without breaking downstream Eventstreams, Eventhouse ingestion, applications, or other consumers. Today, schemas can be versioned, but compatibility enforcement is not yet available. This means producers can introduce changes that may unexpectedly break existing event-processing flows. Fabric should allow teams to define compatibility rules at the schema-set or individual-schema level. Requested capabilities should include: Backward compatibility Forward compatibility Full compatibility No-compatibility mode for intentionally breaking changes Automatic validation before a new schema version is published Clear identification of incompatible field additions, removals, type changes, and structural changes Breaking-change warnings Dependency and impact analysis showing which Eventstreams, Eventhouse tables, destinations, and applications use the affected schema Side-by-side schema diff between versions CI/CD validation API so compatibility checks can run before deployment Approval workflow for intentionally breaking changes Ability to mark older schema versions as deprecated with an effective retirement date For example, if a producer changes a field from integer to string, removes a required property, or changes a nested structure, Fabric should immediately show whether the new version is compatible with existing consumers before the schema is activated. The experience should also show exactly which downstream artifacts may be affected by the change. This would turn Event Schema Registry into a true enterprise contract-management capability rather than simply a centralized place to store schema versions. It would significantly reduce production failures caused by unexpected schema evolution and make Fabric Eventstream safer for large-scale real-time and event-driven architectures.Add Dry-Run Simulation and Historical Event Replay for Fabric Activator Rules
Fabric Activator should provide a dry-run simulation mode that allows developers to test rules against historical or sample event data without executing real downstream actions. Activator rules can trigger operational actions such as notifications, pipelines, workflows, and other automated processes. Before deploying or modifying these rules in production, teams need a safe way to understand exactly how the rule would have behaved against real data. Activator should allow users to select a historical time range, Eventstream data, or sample event set and replay those events through a rule in simulation mode. During simulation, Fabric should evaluate the complete rule logic but suppress external actions. The simulation experience should show: Which events would have matched the rule Which events would not have matched Why each condition evaluated to true or false The exact time each activation would have occurred Which downstream action would have been invoked The parameters that would have been passed to the action Duplicate or repeated activations Suppression, cooldown, and timing behavior Errors or missing fields encountered during evaluation Total simulated activations for the selected period Users should also be able to compare the behavior of the current rule with a modified version before publishing the change. For example, an operations team may change an equipment-temperature rule from 90 degrees to 85 degrees. Before deploying the change, they should be able to replay the previous seven days of telemetry and see how many additional alerts and actions the new rule would have generated. This would make it possible to regression-test Activator rules before production deployment and reduce the risk of alert storms, incorrect pipeline execution, excessive notifications, and unintended automation. A reusable simulation API would also allow teams to include Activator rule validation in automated CI/CD testing.Support Schema Evolution for Fabric Graph Without Rebuilding the Graph Model
Fabric Graph should support controlled schema evolution as enterprise business models change over time. Graph models are rarely static. Organizations need to add new node types, properties, relationships, keys, and source mappings as new business requirements emerge. These changes should be manageable without forcing teams to recreate the graph model and reconnect downstream dependencies. Fabric Graph should support compatible schema changes directly on an existing graph model while preserving the graph identity, security configuration, queries, lineage, and downstream references. Requested capabilities include: Add or modify node properties Add new node and relationship types Add or update source mappings Safely modify keys and relationship definitions Schema version history Compare and diff between graph schema versions Breaking-change detection Dependency and impact analysis before deployment Migration preview Rollback to a previous schema version CI/CD support for promoting graph schema changes across development, test, and production environments For potentially breaking changes, Fabric should identify the affected queries, Digital Twin models, Fabric IQ ontologies, dashboards, Data Agents, and other downstream consumers before the change is applied. For example, an organization may initially model Asset, Site, and Work Order entities and later introduce Manufacturer, Sensor, Maintenance Contract, and additional relationships. These changes should be evolutionary rather than requiring a graph redesign or recreation. Enterprise graph models will continuously evolve. Fabric should provide a governed schema-evolution experience that allows organizations to extend those models safely while protecting existing applications and dependencies.Add Incremental and Event-Driven Refresh for Fabric IQ Ontology
Fabric IQ Ontology should support incremental and event-driven updates when the underlying source data changes. For operational and real-time scenarios, ontology entities and relationships may be backed by Lakehouse, Eventhouse, semantic models, or other Fabric data sources that change frequently. Fabric should not require users to manually refresh or rebuild large portions of the ontology simply to make changed source data available. Fabric IQ should automatically identify which entities, properties, and relationships are affected by source-data changes and update only the impacted portions of the ontology. The capability should include change-aware incremental refresh, Eventstream-triggered refresh, pipeline-triggered refresh, configurable freshness targets, entity-level refresh status, last-successful-update timestamps, failed-record diagnostics, dependency-aware refresh ordering, and monitoring of refresh processing and capacity consumption. For example, if the status of an Asset, Work Order, Shipment, Customer, Device, or other operational entity changes, Fabric IQ should be able to update the relevant ontology state quickly without requiring a full ontology refresh. This would make Fabric IQ significantly more practical for operational intelligence, Digital Twin scenarios, Real-Time Intelligence, and AI agents that depend on current business context.
