eventstream

173 Topics

Real-Time Intelligence in Microsoft Fabric
Join Microsoft Fabric Builders Hub Hyderabad for our first official community event featuring Rajendra Ongole. Discover how Real-Time Intelligence in Microsoft Fabric enables organizations to collect, process, analyze, and act on streaming data with minimal latency. Whether you're a student, beginner, Power BI enthusiast, aspiring data engineer, or early-career professional, this session will introduce the fundamentals of real-time analytics and demonstrate how Microsoft Fabric simplifies building modern, data-driven solutions. In this session, you'll learn about: • What Real-Time Intelligence in Microsoft Fabric is • Key components and architecture • Eventstreams and real-time data ingestion • Building real-time analytics solutions • Practical use cases and demonstrations • Best practices and learning resources • Live Q&A with the speaker Who should attend? • Students • Beginners exploring Microsoft Fabric • Power BI enthusiasts • Aspiring Data Engineers • Data Analysts • Early-career professionals We look forward to welcoming you to the inaugural event of Microsoft Fabric Builders Hub Hyderabad!
VarshiniBuggana
2 days ago Place Microsoft Fabric Builders Hub Hyderabad
37Views
0likes
0Comments
Best practice for handling schema evolution in Fabric Eventstream before data reaches Eventhouse?
I have an Eventstream receiving operational events where the schema may evolve over time. For example, the producer initially sends: DeviceId, Timestamp, Temperature, Status but later adds fields such as: Location, FirmwareVersion, ErrorCode I want the pipeline to continue ingesting events without breaking downstream KQL tables, update policies, materialized views, or Real-Time Dashboards. I am trying to understand where schema evolution should ideally be handled in a production Fabric RTI architecture. Would you: enforce the contract upstream using Schema Registry normalize changing fields inside Eventstream before Eventhouse ingestion land the raw payload first and handle schema evolution inside Eventhouse/KQL maintain separate versioned event schemas/tables How are people handling this in production when producers can add fields without notice? I am particularly interested in avoiding a design where every small upstream schema change forces updates across Eventstream, KQL tables, update policies, and downstream dashboards.
Solved
ShivekMaharaj
3 days ago Place Real-Time Intelligence
80Views
0likes
2Comments
Capacity overview events and Timepoints details
Good morning, is there any update on when the information about the operations provided by the 'Capacity overview events' connector from an eventstream will be included? thanks in advance
rgsalido
5 days ago Place Eventstream
94Views
0likes
3Comments
Best practice for deciding between Eventstream transformations and Eventhouse update policies
Hi Fabric Community, I am exploring a Real-Time Intelligence architecture and would appreciate some guidance on where transformation logic should ideally be placed. The proposed flow is: Azure Event Hubs → Fabric Eventstream → Eventhouse → Real-Time Dashboard / Power BI Fabric Eventstream supports filtering, field management, aggregation and other processing before events are written to the destination. An Eventhouse can also ingest the raw events first and transform them into curated tables through KQL update policies. I am trying to understand the recommended boundary between these two layers. For example, assume the incoming event contains: Device or customer identifier Event timestamp Event type Location Numeric readings Additional JSON properties The required processing includes: Removing events that fail basic validation Renaming and standardizing fields Converting timestamps and data types Flattening selected JSON properties Enriching the event with reference data Creating five-minute aggregates Preserving the original event for auditing and future reprocessing My current thinking is: Use Eventstream for lightweight filtering, routing and simple schema normalization. Land the original event in a Bronze table whenever replay or auditing is required. Use Eventhouse update policies or KQL for enrichment, reusable business logic and curated Silver tables. Use materialized views for frequently queried aggregations rather than calculating them repeatedly in dashboards. However, I am unsure where Microsoft recommends drawing the line. A few questions: Are there transformation types that should generally remain in Eventstream rather than Eventhouse? Is it considered good practice to send both the raw stream and a transformed derived stream into separate Eventhouse tables? When using Eventstream’s Event processing before ingestion mode, what are the trade-offs compared with direct ingestion followed by an Eventhouse update policy? How do teams handle changes to transformation logic when historical events need to be reprocessed? For reference-data enrichment, would you normally perform the lookup in Eventstream or after ingestion with KQL? Are five-minute or hourly aggregations better implemented in Eventstream, through an update policy, or with an Eventhouse materialized view? Microsoft’s Eventstream destination guidance documents both direct ingestion and event processing before ingestion, while the KQL update-policy documentation provides another way to transform ingested data. I would be interested to hear how others divide responsibility between Eventstream and Eventhouse in production, particularly where auditability, reprocessing and maintainability are important. Thanks in advance!
Solved
ShivekMaharaj
10 days ago Place Real-Time Intelligence
90Views
0likes
2Comments
Streaming Architecture
Hi everyone, I'm exploring Microsoft Fabric Real-Time Intelligence and would love to understand how it's being used in enterprise environments. Which business scenarios have benefited the most from real-time analytics? How do you balance: Low latency High throughput Cost Scalability Reliability I'm interested in learning from real production implementations. Thank you!
Solved
binitafulpagare
29 days ago Place Real-Time Intelligence
210Views
0likes
4Comments
From Rain Drops to Playability: Scoring Real-Time Weather Insights for FIFA World Cup 2026⚽
In this blog, I will walk you through how we can support the FIFA World Cup 2026 by organizing teams and players by delivering real-time, actionable weather insights. By predicting critical weather patterns - such as high humidity, extreme heat, and sudden rainfall - teams can proactively adjust their game-day strategies, and organizers can ensure optimal player safety. To bring this vision to life, we will build a comprehensive, real-time weather monitoring dashboard covering all 16 FIFA World Cup stadiums, powered entirely by the real-time analytics capabilities of Microsoft Fabric
suparnababu8
30 days ago Place Real-Time Intelligence Community Blog
381Views
11likes
6Comments
Eventstream connection with Event Hub
Hello, I have an existing event hub which I'm planning to use with an eventstream in Fabric. I see two connection options: 1. SAS keys 2. Organisational account I'm in a microsoft internal team and we're not allowed to use SAS keys. When I try to use my own account via organisational account option I'm getting following error Sign in failed with the following error code: "invalid_request". Error description given: "AADSTS65002: Consent between first party application 'b52893c8-bc2e-47fc-918b-77022b299bbc' and first party resource '80369ed6-5f11-4dd9-bef3-692475845e77' must be configured via preauthorization - applications owned and operated by Microsoft must get approval from the API owner before requesting tokens for that API. Trace ID: 3e344c2e-4ab0-442d-af25-23f3e1c15500 Correlation ID: 18583d2a-007b-4b7a-a7d0-5485087e55c7 Timestamp: 2025-07-29 16:05:32Z" (Session ID: f67ac96c-b700-4aed-8514-2b9acd5b00c8, Region: usfr) I'm wondering how to solve this and also is there any support to use Managed Identities for auth? This is normally possible but I don't see how via Microsoft Fabric.
iremugurlu
1 month ago Place Eventstream
10KViews
1like
4Comments
Options for High‑Frequency, Low‑Latency API Ingestion
I’m looking for guidance on options for invoking external APIs(with dynamic date parameters in specific format like '202602152359') at very high frequency (every 2 minutes or less) and making the data available with minimal end‑to‑end latency. Requirements: Invoke external APIs at 2‑minute intervals (or less). Parse incoming XML/JSON data into structured tables. Refresh user reports/streamlit dashboards quickly, alongside other enterprise data, to support near‑real‑time decision‑making wihtout the delay of sql endpoint refresh. We currently use a Fabric medallion architecture with payload based ingestions. However, this approach is not well‑suited for this use case, as running pipelines/notebooks on 2–5 minute schedules is triggering 70% Fabric capacity alerts.
prathijp
1 month ago Place Eventstream
2.4KViews
1like
7Comments
Heartbeat Action Query Not Sent to Source
I created an Eventstream with Postgre SQL CDC as source. I want to use heartbeat action query to ensure that Eventstream to keep the replication slot size low in source. There is no issue/error in the Eventstream but the heartbeat query is never been sent to source. Is there a configuration I missed? This is the table that I created and the initial value: CREATE TABLE public.debezium_heartbeat ( id SERIAL PRIMARY KEY, timestamp TIMESTAMP WITH TIME ZONE ); INSERT INTO public.debezium_heartbeat (id, timestamp) VALUES (1, now()); This is the heartbeat action query that I put on the Eventstream source: UPDATE public.debezium_heartbeat SET timestamp = now() WHERE id = 1
Solved
lyancy
1 month ago Place Eventstream
1.2KViews
1like
7Comments
Eventhouse vs Lakehouse - When to Use Which in Microsoft Fabric?
Microsoft Fabric offers multiple storage and analytical engines. This blog helps you make the right architectural choice early on and save them massive migration efforts later. In this blog, I’ll walk you through the differences between these two powerhouses, look at real-world challenges, map out concrete scenarios, and give you a definitive comparison framework so you can choose the right tool for the job.
suparnababu8
1 month ago Place Real-Time Intelligence Community Blog
4.9KViews
11likes
5Comments