We’re thrilled to announce that you now have a single place for your streaming data-in-motion in Microsoft Fabric: Real-Time hub (preview)! Real-Time hub enables you to easily discover, manage, and consume data-in-motion from a wide variety of streaming sources.
The hub provides a unified space to access and oversee all streams and events within your organization. It simplifies collaboration, making the development of streaming applications straightforward and centralized. Once your data is streaming in real-time, Real-Time hub empowers you to process, analyze, and leverage insights immediately, transforming data streams into actionable intelligence.
What is Real-Time hub
A. Single place for all data-in-motion - Real-Time hub enables users to easily discover, manage, and consume data-in-motion from a wide variety of sources. It lists all the streams and KQL tables that customers can directly act on.
B. Numerous connectors to simplify data ingestion - Real-Time hub makes it easy for you to ingest data into Fabric from a wide variety of sources like Microsoft streaming sources, Amazon Kinesis Data streams, Confluent Cloud Kafka, Google Cloud Pub/Sub and Fabric events.
C. Real-Time hub is a unified view of all the streams - All data streams in Fabric automatically show up in the hub. Also, users can subscribe to events in Fabric gaining insights into the health and performance of their data ecosystem.
D. Process, Analyze, Visualize and Act on Data streams & Events - Once the streaming data is in Real-Time hub, you can process, analyze and act on the streams.
- Process: You can do real-time processing by using operators such as transform, aggregate, and route your data to desired destination by opening the stream in Fabric Eventstream. The no-code experience in Eventstream allows you to drag and drop to design the event data processing logic while using temporal window functions.
- Analyze: You can send the data streams with/without transformation to KQL Database where you can run KQL queries to immediately analyze & visualize the data.
- Act: You can set an alert on data streams & events by setting up a Reflex item. Reflex from Data Activator helps you to set alerts based on conditions and specify actions to take when the conditions are met.
Components of Real-Time hub
There are three tabs in the Hub:
- Data streams
- This tab contains all real-time streams in Fabric that the user has access to. This includes all streams from Eventstreams and all tables from KQL Databases.
- Microsoft sources
- This tab makes it super easy for you to discover Microsoft resources from which you can bring streaming data into Fabric. The tab displays all Microsoft sources that user has access to. You can then connect to a source and bring the data streams in Fabric.
- Fabric events
- A number of events are now surfaced from Fabric artifacts and Azure resources in the Real-Time hub to enable event-driven flows. For example, you can react to events from Azure Storage such as when a new file is created in a particular folder.
Let’s dive into each tab stated above in detail:
Data streams
On this tab, you will see all streams from Eventstreams and all tables from KQL Databases in Fabric. The Hub makes it easy to find the stream you are looking for by allowing you to do free-text search or by using the Owner, Item name and Location filters. Each stream has three actions that a user can take. Those actions allow you to easily preview incoming data, open the stream or endorse the stream. These actions help in understanding a data stream end to end and to make it more discoverable.
Clicking on a stream brings you to a detailed view of the stream where you can view the data that’s flowing through the selected stream. Also, monitor the number of messages that are incoming and outgoing through that selected stream. In addition to monitoring the data in the stream, view the upstream sources and downstream destinations of a selected stream.
For the selected stream, you can directly set an alert on the stream or open the Eventstream for further editing. An alert can be set up by creating a Reflex item on the stream. Reflex from Data Activator helps you to set alerts based on conditions and specify actions to take when the conditions are met.
Microsoft sources
On this tab, you will see the many types of Microsoft sources that you have access to. The following types of Microsoft sources are supported:
- Azure Event Hubs
- Azure IoT Hub
You can connect to the following databases hosted in Azure that you can stream change data capture feeds:
- Azure SQL Database CDC
- Azure Database for PostgreSQL CDC
- Azure Database for MySQL CDC
- Azure Cosmos DB CDC
The hub helps narrow down the list of Microsoft resources by providing a set of filters for source type, Azure subscription, Azure resource group, and region. You can then connect to a resource to bring streaming data into Fabric with an Eventstream.
Fabric events
This tab allows you to monitor and react to the following events:
- Azure Blob Storage events: Events produced by Azure storage accounts
- Fabric workspace item events: Events produced by the Fabric platform when there’s activity in a workspace
Azure Blob Storage events: Azure blob storage events allow you to receive notifications when certain Activities occur in the storage account. For example, you can receive a notification when a new blob is created, or an existing blob is modified. These events can be used to trigger downstream actions or workflows, such as running a data pipeline, sending email notifications etc.
Fabric workspace item events: By using Fabric workspace item events, you can receive notifications when activity occurs in your workspace such as new item is created or deleted. These events can be used to set alerts using Data Activator to run data pipelines, send email or notifications on Teams.
Learn more, and help us with your feedback
To learn more about Real-Time hub, please visit https://learn.microsoft.com/fabric/real-time-hub/
To find out more about Real-Time Intelligence, read Yitzhak Kesselman’s announcement. As we launch our preview, we’d love to hear what you think and how you’re using the product. The best way to get in touch with us is through our community forum or submit an idea. For detailed how-tos, tutorials and other resources, check out the documentation.
This is part of a series of blog posts that dive into all the capabilities of Real-Time Intelligence. Stay tuned for more!