Enhanced Eventstream and the connector sources is now generally available!
Enhanced Eventstream GA
Enhanced Eventstream introduces new enhancement features that improve building stream flows in Fabric Real-Time Intelligence. Data engineers can now leverage Edit and Live View modes, Default and Derived Streams, and Smart Routing offering greater flexibility and efficiency
- Edit Mode and Live View: Eventstream now offers two separate modes to give you flexibility and control over your data streams. Edit mode allows you design and modify your data streaming flow without interrupting the live data streams. Live View gives real-time insight into the data flow, allowing you to monitor the ingestion, processing, and distribution of data streams within the Fabric. You can switch between the two modes using the button in the top-right corner. To learn more, refer to our documentation, Edit and publish Microsoft Fabric eventstreams.
- Default and Derived Streams: Data stream is a dynamic and continuous flow of data that allows you to set up real-time alerts, and feed into different types of data stores. Default stream is created automatically when a streaming source is added to Eventstream, capturing raw event data directly from the source and preparing it for transformation or analysis. Derived stream is a specialized stream that users can set up as a destination within Eventstream. After performing operations such as filtering and aggregating, the derived stream is ready for further analysis or consumption by other organization members through the Real-Time hub. To learn more, refer to our documentation, Create default and derived Fabric eventstreams.
- Content-based Routing: Users are now able to design stream operations directly within Eventstream’s Edit mode, transforming and routing of real-time data streams. It lets you create stream processing logic and direct data streams based on their content right in the Eventstream editor. To learn more, refer to our documentation, Route events based on content in Fabric event streams.
Eventstream Connector Sources GA
The new connector sources in enhanced Eventstream enable seamless connection of external real-time data streams to Fabric, allowing for an optimal out-of-the-box experience and more choices for real-time insights from a variety of sources. It supports well-known cloud services like Google Cloud and Amazon Kinesis, as well as database change data capture (CDC) streams through our new messaging connectors. These connectors utilize Kafka connect and Camel Kafka connectors for a flexible approach to data integration, ensuring broad connectivity across leading platforms. Additionally, Debezium is integrated for precise CDC stream capture.
Generally Available Eventstream Sources
- Confluent Cloud Kafka
- Amazon Kinesis Data Streams
- Google Cloud Pub/Sub
- Amazon MSK Kafka
- Azure SQL Database Change Data Capture (CDC)
- Azure SQL Managed Instance (CDC)
- SQL Server on VM DB (CDC)
- PostgreSQL DB (CDC)
- Azure Cosmos DB (CDC)
- MySQL DB (CDC)
To learn more about sources regarding the details of the configuration, refer to our documentation Add and manage eventstream sources.
To ask for new connector sources, please contact us at askeventstreams@microsoft.com, or share your preferred sources in our survey: https://aka.ms/EventStreamSources.
Data Preview on Database CDC Connector Sources in Eventstream
The enhanced Eventstream now offers Data Preview for database CDC sources. This feature lets you view data snapshots in both Edit and Live View modes. In Edit mode, you can capture a snapshot from your configured CDC source to infer the schema for subsequent operators or destinations without publishing first. Access the data preview in the 'Test result' tab at the bottom pane by selecting the source node on the canvas in Edit mode.
Similarly in Live View mode, the data preview provides you with a snapshot from your sources so that you can understand what the data looks like inside your sources.
Supported connector sources:
- Azure SQL Database Change Data Capture (CDC)
- Azure SQL Managed Instance (CDC)
- SQL Server on VM DB (CDC)
- PostgreSQL DB (CDC)
- Azure Cosmos DB (CDC)
- MySQL DB (CDC)
Monitoring Experience on Connector Sources with Runtime Logs and Data Insights
Eventstream now provides Runtime Logs and Data Insights for connector sources in Live View mode. Runtime Logs allow you to review detailed logs produced by the connector engines for a specific connector, aiding in the identification of failure causes or warnings. This feature can be accessed in the bottom pane of Eventstream by selecting the relevant connector source node on the canvas in Live View mode.
Data Insights offers metrics from the connector engine, helping users monitor the status and performance of connector sources. The Source Incoming/Outgoing Events show the records polled or produced by tasks assigned to the source connector in the worker.
To learn more, refer to our documentation, Monitoring status and performance of an Eventstream item.
Three Additional Preview Features
In addition to these generally available features highlighted above, there are three new features that are open for preview now.
Introducing Azure Service Bus Connector Source in Eventstream
Many enterprise customers use Azure Service Bus for managing queues and publish-subscribe topics. They seek to integrate this with Fabric for seamless data streaming, high-performance processing, and real-time dashboards.
We're excited to announce the Azure Service Bus connector source for Eventstream! This connector streams messages from Azure Service Bus topics and queues directly into Eventstream, enabling real-time processing and routing to multiple destinations within Fabric. It simplifies integration and enhances scalable data streaming from your Azure messaging sources.
The screenshot below shows the Azure Service Bus source tile in wizard and the configuration parameters in Eventstream's edit mode.
To learn more about the configuration guide details for this source, refer to our documentation, Add Azure Service Bus source to an eventstream.
Introducing New Fabric Events Categories in Eventstream Source
Eventstream enables the capture of Fabric events in various categories, their transformation, and routing to various destinations within Fabric for further analysis. We are pleased to announce the introduction of two new categories of Fabric events: Job events and OneLake events. These event categories are now available in Eventstream source, allowing you to capture and process these events within Eventstream.
- Job events: These are events generated when jobs are updated within Fabric. For example, they may include updates related to a data pipeline or notebook execution. The events can encompass details on job initiation, completion, failure, and other relevant statuses.
- OneLake events: Events produced when changes occur in your selected OneLake folders. For example, a new file or folder is created or deleted in the selected folder.
To use the two sources within Eventstream, select 'Use external source' to open the source wizard and make your selection.
To learn more about how to add and configure these sources, refer to our documentation Add and manage eventstream sources.
Securely Stream Data to Eventstream with Entra ID Authentication
Entra ID authentication for Eventstream’s Custom Endpoint is introduced! This feature secures data streaming without using SAS keys or connection strings, reducing unauthorized access risk. Entra ID authentication links user permissions to Fabric workspace access, ensuring only authorized users can stream data to Eventstream.
Furthermore, Tenant Admins now have the capability to disable key-based authentication for Eventstream within tenant settings, thereby enhancing security by mandating the use of Entra ID authentication exclusively.
The feature will soon be available for preview.
What’s next?
There will be many more exciting developments as we continue to innovate and expand the capabilities of Real-Time Intelligence. Learn more about all the features and follow a step-by-step tutorial. Join the conversation and vote for your favorite features.
Over the next month we’ll be releasing a series of blog posts that dive into all the capabilities further. Stay tuned for more!