eventstream

166 Topics

Edge device detections into Fabric: Eventstream + Eventhouse, or Lakehouse?
Hi all, While working on a computer vision project at DataToBiz, we built the edge side of a real-time system, and I’d like to compare notes with people here on how the data side would work in Fabric. On the edge, the detection runs on a Raspberry Pi. We used YOLOv8n with ByteTrack to track vehicles across frames, plus Depth-Anything-v2 for depth estimation from a single camera so we could apply spacing rules. The model was converted to ONNX, TFLite and NCNN, and we used quantization and threading to get the speed we needed without heavy hardware. A camera, GPS and compass were synced by timestamp, so every detection comes out tagged with its time, location and direction. Post-processing smooths the detections to reduce flickering. That leaves a stream of time-stamped, geo-tagged detection events from many devices, and that’s the part I’d like your input on in Fabric. Would you ingest it through Eventstreams into an Eventhouse and query it with KQL, or land it in a Lakehouse and process it in batches? Where does that line sit for you when the end goal is near real-time dashboards? How do you handle devices that go offline and send their events late or out of order? Do you rely on event time in the Eventhouse, or clean it up downstream? For the geo side, with coordinates and direction on every event, how have you handled spatial queries and map visuals in Real-Time Intelligence or Power BI? Is it good enough, or do you still end up pre-processing the coordinates? And with raw detections arriving continuously, how do you decide what to keep as raw events, and what to aggregate into something like per-location availability for reporting? Would love to hear how you’d approach it.
Solved
kiara-mehra35
7 days ago Place Real-Time Intelligence
87Views
0likes
4Comments
Handling late and out-of-order events in Eventstream windows and Eventhouse aggregations?
I'm coming from Apache Flink, where late and out-of-order events are handled with event-time watermarks and allowed lateness. I'm trying to understand the equivalent approach in Fabric Real-Time Intelligence. Scenario: IoT devices send telemetry with an event timestamp. Because of network delays, some events arrive several minutes late, and occasionally out of order. I need accurate 5-minute aggregates (average and max per device) for dashboards and Activator rules. Questions: Eventstream: when using windowed aggregations (tumbling or hopping), are windows based on event time or arrival time? Is there a setting for out-of-order tolerance or late-arrival handling, and what happens to events that arrive after a window has closed: are they dropped, or do they produce a corrected result? Eventhouse: if I aggregate in Eventhouse instead, for example with a materialized view using bin(EventTime, 5m), are late events included automatically when they arrive? Any caveats with materialized views and late data? Which layer is better for aggregations when correctness under late data matters more than lowest latency? Activator: if a rule has already fired on an aggregate that later gets corrected by late data, is there a pattern for handling that, or should rules only run on finalized windows?
Solved
jubinsoni
11 days ago Place Real-Time Intelligence
86Views
1like
6Comments
Best practice for handling schema evolution in Fabric Eventstream before data reaches Eventhouse?
I have an Eventstream receiving operational events where the schema may evolve over time. For example, the producer initially sends: DeviceId, Timestamp, Temperature, Status but later adds fields such as: Location, FirmwareVersion, ErrorCode I want the pipeline to continue ingesting events without breaking downstream KQL tables, update policies, materialized views, or Real-Time Dashboards. I am trying to understand where schema evolution should ideally be handled in a production Fabric RTI architecture. Would you: enforce the contract upstream using Schema Registry normalize changing fields inside Eventstream before Eventhouse ingestion land the raw payload first and handle schema evolution inside Eventhouse/KQL maintain separate versioned event schemas/tables How are people handling this in production when producers can add fields without notice? I am particularly interested in avoiding a design where every small upstream schema change forces updates across Eventstream, KQL tables, update policies, and downstream dashboards.
Solved
ShivekMaharaj
15 days ago Place Real-Time Intelligence
172Views
0likes
3Comments
Best pattern for keeping a current-state table in Eventhouse from Eventstream CDC?
Hi all, I'm streaming change data from Azure SQL Database into Eventhouse using an Eventstream CDC source. Ingesting the raw change events works fine. The question is how best to turn them into a queryable current-state table, the way you would with a Flink CDC or Debezium pipeline. Setup: Source: Azure SQL Database (CDC enabled), about 20 tables, mostly updates with occasional deletes Eventstream → Eventhouse, raw change events land in one table per source table Consumers: Real-Time Dashboards and some Activator rules that need the latest row per key What I'm considering: A materialized view using arg_max(timestamp, *) by key over the raw table An update policy that flattens the change envelope into a typed table, with the materialized view on top of that Handling deletes by keeping a delete flag, then filtering it out in the view or a stored function Questions: Is option 2 (update policy, then materialized view) the recommended pattern, or is there a simpler built-in approach? How are people handling deletes? Filtering the latest row by a delete flag works, but deleted keys keep taking up space. Is there a good retention or cleanup pattern? What should I use as the ordering column so out-of-order events don't produce a stale "latest" row: the source LSN/commit timestamp, or ingestion time? Any gotchas when the source schema changes (for example, a new column)? Thanks!
Solved
jubinsoni
27 days ago Place Real-Time Intelligence
89Views
1like
4Comments
Date slicer
Hi, I need some help with Power BI. I want to use a Between Date Slicer and display the dates in this format: DD-MMM-YYYY Example: 01-Jan-2020 I don’t want to use the default dropdown/date format. I specifically want the Between slicer with two date inputs (From Date and To Date), but I need the displayed dates to appear as 01-Jan-2020 instead of the default format. Could you please guide me on how to achieve this? If the standard Power BI slicer does not support this format, is there any custom visual or alternative solution that can provide the same Between Date functionality with the DD-MMM-YYYY format? Thanks!
Solved
pradipir
1 month ago Place Real-Time Intelligence
81Views
1like
5Comments
Fabric Business Events: what delivery guarantees and replay pattern should we design for?
Hi all, I am testing the newer Business Events capability in Fabric Real-Time Intelligence and trying to understand what reliability assumptions should be made for a production design. The pattern I am looking at is roughly: Eventstream → Business Event → Activator → downstream action / User Data Function with Eventhouse enabled so the published business events are also retained for historical analysis. The current documentation explains the publisher/consumer model and shows how Eventstream can publish a governed business event that Activator then consumes. What I have not been able to find clearly documented is the delivery contract between the published business event and its consumers. A few things I am trying to clarify: If an Activator consumer or downstream action is temporarily unavailable, does Fabric retry delivery of the business event? Should consumers assume at-least-once delivery and therefore be designed to handle duplicate events, or is a different delivery model used? Is event ordering guaranteed in any scope, for example for events from the same Eventstream publisher? Since published business events can also be retained automatically in Eventhouse, is that retained history intended to support replay/reprocessing after a consumer outage, or is it primarily an analytical record and replay would need to be implemented separately? Are there documented retry or delivery-retention windows that should be considered when designing an operational workflow? I am mainly trying to understand what a resilient production pattern should look like when the business event triggers something with side effects, where processing the same event twice or silently missing an event would matter. Would you generally make the downstream consumer idempotent and treat Eventhouse as an audit/recovery store, or is there a more Fabric-native pattern for this? Interested to hear how others are approaching this with Business Events and Activator.
ShivekMaharaj
2 months ago Place Real-Time Intelligence
132Views
0likes
3Comments
Capacity overview events and Timepoints details
Good morning, is there any update on when the information about the operations provided by the 'Capacity overview events' connector from an eventstream will be included? thanks in advance
Solved
rgsalido
2 months ago Place Eventstream
224Views
0likes
7Comments
Best practice for deciding between Eventstream transformations and Eventhouse update policies
Hi Fabric Community, I am exploring a Real-Time Intelligence architecture and would appreciate some guidance on where transformation logic should ideally be placed. The proposed flow is: Azure Event Hubs → Fabric Eventstream → Eventhouse → Real-Time Dashboard / Power BI Fabric Eventstream supports filtering, field management, aggregation and other processing before events are written to the destination. An Eventhouse can also ingest the raw events first and transform them into curated tables through KQL update policies. I am trying to understand the recommended boundary between these two layers. For example, assume the incoming event contains: Device or customer identifier Event timestamp Event type Location Numeric readings Additional JSON properties The required processing includes: Removing events that fail basic validation Renaming and standardizing fields Converting timestamps and data types Flattening selected JSON properties Enriching the event with reference data Creating five-minute aggregates Preserving the original event for auditing and future reprocessing My current thinking is: Use Eventstream for lightweight filtering, routing and simple schema normalization. Land the original event in a Bronze table whenever replay or auditing is required. Use Eventhouse update policies or KQL for enrichment, reusable business logic and curated Silver tables. Use materialized views for frequently queried aggregations rather than calculating them repeatedly in dashboards. However, I am unsure where Microsoft recommends drawing the line. A few questions: Are there transformation types that should generally remain in Eventstream rather than Eventhouse? Is it considered good practice to send both the raw stream and a transformed derived stream into separate Eventhouse tables? When using Eventstream’s Event processing before ingestion mode, what are the trade-offs compared with direct ingestion followed by an Eventhouse update policy? How do teams handle changes to transformation logic when historical events need to be reprocessed? For reference-data enrichment, would you normally perform the lookup in Eventstream or after ingestion with KQL? Are five-minute or hourly aggregations better implemented in Eventstream, through an update policy, or with an Eventhouse materialized view? Microsoft’s Eventstream destination guidance documents both direct ingestion and event processing before ingestion, while the KQL update-policy documentation provides another way to transform ingested data. I would be interested to hear how others divide responsibility between Eventstream and Eventhouse in production, particularly where auditability, reprocessing and maintainability are important. Thanks in advance!
Solved
ShivekMaharaj
2 months ago Place Real-Time Intelligence
134Views
0likes
2Comments
Streaming Architecture
Hi everyone, I'm exploring Microsoft Fabric Real-Time Intelligence and would love to understand how it's being used in enterprise environments. Which business scenarios have benefited the most from real-time analytics? How do you balance: Low latency High throughput Cost Scalability Reliability I'm interested in learning from real production implementations. Thank you!
Solved
binitafulpagare
3 months ago Place Real-Time Intelligence
261Views
0likes
4Comments
Eventstream connection with Event Hub
Hello, I have an existing event hub which I'm planning to use with an eventstream in Fabric. I see two connection options: 1. SAS keys 2. Organisational account I'm in a microsoft internal team and we're not allowed to use SAS keys. When I try to use my own account via organisational account option I'm getting following error Sign in failed with the following error code: "invalid_request". Error description given: "AADSTS65002: Consent between first party application 'b52893c8-bc2e-47fc-918b-77022b299bbc' and first party resource '80369ed6-5f11-4dd9-bef3-692475845e77' must be configured via preauthorization - applications owned and operated by Microsoft must get approval from the API owner before requesting tokens for that API. Trace ID: 3e344c2e-4ab0-442d-af25-23f3e1c15500 Correlation ID: 18583d2a-007b-4b7a-a7d0-5485087e55c7 Timestamp: 2025-07-29 16:05:32Z" (Session ID: f67ac96c-b700-4aed-8514-2b9acd5b00c8, Region: usfr) I'm wondering how to solve this and also is there any support to use Managed Identities for auth? This is normally possible but I don't see how via Microsoft Fabric.
iremugurlu
3 months ago Place Eventstream
10KViews
1like
4Comments