Get data from Apache Kafka topics via private networking
Current prerequisites limit to public Kafka instances which is not feasible for most enterprise customers. When will this limitation be lifted so that it can be used in private network setups, for example via gateway? https://learn.microsoft.com/en-gb/fabric/real-time-intelligence/event-streams/add-source-apache-kafka#prerequisites
NL to KQL/GQL generation
Hi Team, This is coming from Using Data Agents for NL to KQL/GQL generation | Microsoft Fabric Community. It would be great if we could have a feature where the data agent or the ontology agent can return just the grounded KQL/GQL query (or query variants) required to answer a NL question rather than actually execute it and summarize the result. This is important in use-cases like ours where retrieval needs to be split from query generation. Currently Fabric IQ doesn't provide a way to do that, and the only way seems to be to inspect the data agent's run steps which seems roundabout for something that might be asy to support natively. Please consider this feature request. Thanks!Support Creator-Independent Managed Identity and Ownership Transfer for Fabric Operations Agents
Microsoft Fabric Operations Agents should support an enterprise identity and ownership model that is independent of the individual user who originally created the agent. Today, an Operations Agent receives its own Microsoft Entra Agent ID, but the agent operates using the delegated identity and permissions of its creator. This creates an operational dependency on an individual employee account. For production environments, an Operations Agent may run continuously for months or years. The agent should not stop working or unexpectedly lose access because the original creator changes roles, loses access to a workspace, leaves the organization, or has permissions modified. Fabric should support: Creator-independent agent identity Managed identity or service-principal-style execution Explicit permissions assigned directly to the agent Transfer of agent ownership to another user or team Multiple agent owners or administrators Separation between agent owner, operator, approver, and notification recipient Ability to rotate or replace the execution identity without recreating the agent Clear visibility into which identity is used for every query and action Permission validation before an agent is started Alerts when the agent's required permissions are removed Audit history for identity and ownership changes Support for deployment across development, test, and production using environment-specific identities For example, a manufacturing Operations Agent may continuously monitor equipment telemetry and invoke a Fabric notebook or Power Automate workflow when abnormal conditions are detected. If the employee who originally created that agent leaves the project, the production agent should continue operating using an enterprise-managed identity rather than depending on the former employee's delegated permissions. Administrators should also be able to transfer ownership without recreating the agent, regenerating its playbook, or breaking its rules and downstream actions. This capability would make Operations Agents significantly safer for long-running enterprise automation and would align agent lifecycle management with established enterprise security, least-privilege, and service-identity practices.Add Schema Compatibility Policies and Breaking-Change Detection to Fabric Event Schema Registry
Fabric Event Schema Registry should support configurable schema compatibility policies so teams can safely evolve event contracts without breaking downstream Eventstreams, Eventhouse ingestion, applications, or other consumers. Today, schemas can be versioned, but compatibility enforcement is not yet available. This means producers can introduce changes that may unexpectedly break existing event-processing flows. Fabric should allow teams to define compatibility rules at the schema-set or individual-schema level. Requested capabilities should include: Backward compatibility Forward compatibility Full compatibility No-compatibility mode for intentionally breaking changes Automatic validation before a new schema version is published Clear identification of incompatible field additions, removals, type changes, and structural changes Breaking-change warnings Dependency and impact analysis showing which Eventstreams, Eventhouse tables, destinations, and applications use the affected schema Side-by-side schema diff between versions CI/CD validation API so compatibility checks can run before deployment Approval workflow for intentionally breaking changes Ability to mark older schema versions as deprecated with an effective retirement date For example, if a producer changes a field from integer to string, removes a required property, or changes a nested structure, Fabric should immediately show whether the new version is compatible with existing consumers before the schema is activated. The experience should also show exactly which downstream artifacts may be affected by the change. This would turn Event Schema Registry into a true enterprise contract-management capability rather than simply a centralized place to store schema versions. It would significantly reduce production failures caused by unexpected schema evolution and make Fabric Eventstream safer for large-scale real-time and event-driven architectures.Add Schema Compatibility Policies and Breaking-Change Detection to Fabric Event Schema Registry
Fabric Event Schema Registry should support configurable schema compatibility policies so teams can safely evolve event contracts without breaking downstream Eventstreams, Eventhouse ingestion, applications, or other consumers. Today, schemas can be versioned, but compatibility enforcement is not yet available. This means producers can introduce changes that may unexpectedly break existing event-processing flows. Fabric should allow teams to define compatibility rules at the schema-set or individual-schema level. Requested capabilities should include: Backward compatibility Forward compatibility Full compatibility No-compatibility mode for intentionally breaking changes Automatic validation before a new schema version is published Clear identification of incompatible field additions, removals, type changes, and structural changes Breaking-change warnings Dependency and impact analysis showing which Eventstreams, Eventhouse tables, destinations, and applications use the affected schema Side-by-side schema diff between versions CI/CD validation API so compatibility checks can run before deployment Approval workflow for intentionally breaking changes Ability to mark older schema versions as deprecated with an effective retirement date For example, if a producer changes a field from integer to string, removes a required property, or changes a nested structure, Fabric should immediately show whether the new version is compatible with existing consumers before the schema is activated. The experience should also show exactly which downstream artifacts may be affected by the change. This would turn Event Schema Registry into a true enterprise contract-management capability rather than simply a centralized place to store schema versions. It would significantly reduce production failures caused by unexpected schema evolution and make Fabric Eventstream safer for large-scale real-time and event-driven architectures.Add Dry-Run Simulation and Historical Event Replay for Fabric Activator Rules
Fabric Activator should provide a dry-run simulation mode that allows developers to test rules against historical or sample event data without executing real downstream actions. Activator rules can trigger operational actions such as notifications, pipelines, workflows, and other automated processes. Before deploying or modifying these rules in production, teams need a safe way to understand exactly how the rule would have behaved against real data. Activator should allow users to select a historical time range, Eventstream data, or sample event set and replay those events through a rule in simulation mode. During simulation, Fabric should evaluate the complete rule logic but suppress external actions. The simulation experience should show: Which events would have matched the rule Which events would not have matched Why each condition evaluated to true or false The exact time each activation would have occurred Which downstream action would have been invoked The parameters that would have been passed to the action Duplicate or repeated activations Suppression, cooldown, and timing behavior Errors or missing fields encountered during evaluation Total simulated activations for the selected period Users should also be able to compare the behavior of the current rule with a modified version before publishing the change. For example, an operations team may change an equipment-temperature rule from 90 degrees to 85 degrees. Before deploying the change, they should be able to replay the previous seven days of telemetry and see how many additional alerts and actions the new rule would have generated. This would make it possible to regression-test Activator rules before production deployment and reduce the risk of alert storms, incorrect pipeline execution, excessive notifications, and unintended automation. A reusable simulation API would also allow teams to include Activator rule validation in automated CI/CD testing.Support Schema Evolution for Fabric Graph Without Rebuilding the Graph Model
Fabric Graph should support controlled schema evolution as enterprise business models change over time. Graph models are rarely static. Organizations need to add new node types, properties, relationships, keys, and source mappings as new business requirements emerge. These changes should be manageable without forcing teams to recreate the graph model and reconnect downstream dependencies. Fabric Graph should support compatible schema changes directly on an existing graph model while preserving the graph identity, security configuration, queries, lineage, and downstream references. Requested capabilities include: Add or modify node properties Add new node and relationship types Add or update source mappings Safely modify keys and relationship definitions Schema version history Compare and diff between graph schema versions Breaking-change detection Dependency and impact analysis before deployment Migration preview Rollback to a previous schema version CI/CD support for promoting graph schema changes across development, test, and production environments For potentially breaking changes, Fabric should identify the affected queries, Digital Twin models, Fabric IQ ontologies, dashboards, Data Agents, and other downstream consumers before the change is applied. For example, an organization may initially model Asset, Site, and Work Order entities and later introduce Manufacturer, Sensor, Maintenance Contract, and additional relationships. These changes should be evolutionary rather than requiring a graph redesign or recreation. Enterprise graph models will continuously evolve. Fabric should provide a governed schema-evolution experience that allows organizations to extend those models safely while protecting existing applications and dependencies.Add Incremental and Event-Driven Refresh for Fabric IQ Ontology
Fabric IQ Ontology should support incremental and event-driven updates when the underlying source data changes. For operational and real-time scenarios, ontology entities and relationships may be backed by Lakehouse, Eventhouse, semantic models, or other Fabric data sources that change frequently. Fabric should not require users to manually refresh or rebuild large portions of the ontology simply to make changed source data available. Fabric IQ should automatically identify which entities, properties, and relationships are affected by source-data changes and update only the impacted portions of the ontology. The capability should include change-aware incremental refresh, Eventstream-triggered refresh, pipeline-triggered refresh, configurable freshness targets, entity-level refresh status, last-successful-update timestamps, failed-record diagnostics, dependency-aware refresh ordering, and monitoring of refresh processing and capacity consumption. For example, if the status of an Asset, Work Order, Shipment, Customer, Device, or other operational entity changes, Fabric IQ should be able to update the relevant ontology state quickly without requiring a full ontology refresh. This would make Fabric IQ significantly more practical for operational intelligence, Digital Twin scenarios, Real-Time Intelligence, and AI agents that depend on current business context.
Expose full activity-level error details in Workspace Monitoring Eventhouse
Microsoft Fabric Workspace Monitoring exposes activity-level pipeline telemetry in FabricDataPipelinesActivityRunsLogs, including PipelineRunId, OperationId, ActivityIterationCount, ActivityName, ActivityType, Status and FailureType, however, it does not expose the full error information shown in the Fabric Monitoring Hub UI for the same failed activity. The Eventhouse log contains only a generic failure category such as UserError, while the UI can show the actual SQL, notebook, copy, dataflow or semantic-model error message, this prevents reliable centralized operational monitoring.. The Job Scheduler REST API exposes failureReason only at pipeline-run level but that message cannot be safely attributed to a single activity when a pipeline contains parallel branches or more than one failed activity. Please expose a supported, queryable activity-level diagnostics payload in Workspace Monitoring, either directly in the table FabricDataPipelinesActivityRunsLogs or in a documented related table/API. Requested fields: - ErrorCode - ErrorMessage - ErrorDetails or raw diagnostic payload - IsRetriable, where applicable - Correlation to PipelineRunId, OperationId and ActivityIterationCount The payload should be accessible through KQL and preserve the exact association with the failed activity instance, including retry and loop iterations. The Fabric Monitoring Hub already displays these details, so exposing them in the monitoring telemetry would allow organizations to build reliable central monitoring without adding custom OnFailure logging branches to every activity in every pipeline. Business impact: We operate a large Fabric Data Factory estate with existing and future pipelines. A central Eventhouse-based monitoring solution is required for operations and troubleshooting. Without activity-level error details, support teams must manually open each pipeline run in the Fabric UI. Custom per-activity failure handlers are not a scalable alternative because they materially increase the complexity and maintenance cost of every pipeline definition.
