eventstream
174 TopicsFabric Business Events: what delivery guarantees and replay pattern should we design for?
Hi all, I am testing the newer Business Events capability in Fabric Real-Time Intelligence and trying to understand what reliability assumptions should be made for a production design. The pattern I am looking at is roughly: Eventstream → Business Event → Activator → downstream action / User Data Function with Eventhouse enabled so the published business events are also retained for historical analysis. The current documentation explains the publisher/consumer model and shows how Eventstream can publish a governed business event that Activator then consumes. What I have not been able to find clearly documented is the delivery contract between the published business event and its consumers. A few things I am trying to clarify: If an Activator consumer or downstream action is temporarily unavailable, does Fabric retry delivery of the business event? Should consumers assume at-least-once delivery and therefore be designed to handle duplicate events, or is a different delivery model used? Is event ordering guaranteed in any scope, for example for events from the same Eventstream publisher? Since published business events can also be retained automatically in Eventhouse, is that retained history intended to support replay/reprocessing after a consumer outage, or is it primarily an analytical record and replay would need to be implemented separately? Are there documented retry or delivery-retention windows that should be considered when designing an operational workflow? I am mainly trying to understand what a resilient production pattern should look like when the business event triggers something with side effects, where processing the same event twice or silently missing an event would matter. Would you generally make the downstream consumer idempotent and treat Eventhouse as an audit/recovery store, or is there a more Fabric-native pattern for this? Interested to hear how others are approaching this with Business Events and Activator.88Views0likes3CommentsHandling Multiple Schemas in Eventstream for Azure Event Hub and Azure IOT Hub data sources
One of the challenges with streaming data is that data comes in a variety of schemas, which can be dynamic and are not always predictable. As applications and data structures change, schema values can sometimes be wildly different across devices or event inputs. Recently a customer reached out with several questions on this common problem. With the capabilities in Fabric Real Time Intelligence we can flexibly ingest this data. Enough of the background let’s get to the problem!Real-Time Intelligence in Microsoft Fabric
Join Microsoft Fabric Builders Hub Hyderabad for our first official community event featuring Rajendra Ongole. Discover how Real-Time Intelligence in Microsoft Fabric enables organizations to collect, process, analyze, and act on streaming data with minimal latency. Whether you're a student, beginner, Power BI enthusiast, aspiring data engineer, or early-career professional, this session will introduce the fundamentals of real-time analytics and demonstrate how Microsoft Fabric simplifies building modern, data-driven solutions. In this session, you'll learn about: • What Real-Time Intelligence in Microsoft Fabric is • Key components and architecture • Eventstreams and real-time data ingestion • Building real-time analytics solutions • Practical use cases and demonstrations • Best practices and learning resources • Live Q&A with the speaker Who should attend? • Students • Beginners exploring Microsoft Fabric • Power BI enthusiasts • Aspiring Data Engineers • Data Analysts • Early-career professionals We look forward to welcoming you to the inaugural event of Microsoft Fabric Builders Hub Hyderabad!Best practice for handling schema evolution in Fabric Eventstream before data reaches Eventhouse?
I have an Eventstream receiving operational events where the schema may evolve over time. For example, the producer initially sends: DeviceId, Timestamp, Temperature, Status but later adds fields such as: Location, FirmwareVersion, ErrorCode I want the pipeline to continue ingesting events without breaking downstream KQL tables, update policies, materialized views, or Real-Time Dashboards. I am trying to understand where schema evolution should ideally be handled in a production Fabric RTI architecture. Would you: enforce the contract upstream using Schema Registry normalize changing fields inside Eventstream before Eventhouse ingestion land the raw payload first and handle schema evolution inside Eventhouse/KQL maintain separate versioned event schemas/tables How are people handling this in production when producers can add fields without notice? I am particularly interested in avoiding a design where every small upstream schema change forces updates across Eventstream, KQL tables, update policies, and downstream dashboards.Solved107Views0likes2CommentsBest practice for deciding between Eventstream transformations and Eventhouse update policies
Hi Fabric Community, I am exploring a Real-Time Intelligence architecture and would appreciate some guidance on where transformation logic should ideally be placed. The proposed flow is: Azure Event Hubs → Fabric Eventstream → Eventhouse → Real-Time Dashboard / Power BI Fabric Eventstream supports filtering, field management, aggregation and other processing before events are written to the destination. An Eventhouse can also ingest the raw events first and transform them into curated tables through KQL update policies. I am trying to understand the recommended boundary between these two layers. For example, assume the incoming event contains: Device or customer identifier Event timestamp Event type Location Numeric readings Additional JSON properties The required processing includes: Removing events that fail basic validation Renaming and standardizing fields Converting timestamps and data types Flattening selected JSON properties Enriching the event with reference data Creating five-minute aggregates Preserving the original event for auditing and future reprocessing My current thinking is: Use Eventstream for lightweight filtering, routing and simple schema normalization. Land the original event in a Bronze table whenever replay or auditing is required. Use Eventhouse update policies or KQL for enrichment, reusable business logic and curated Silver tables. Use materialized views for frequently queried aggregations rather than calculating them repeatedly in dashboards. However, I am unsure where Microsoft recommends drawing the line. A few questions: Are there transformation types that should generally remain in Eventstream rather than Eventhouse? Is it considered good practice to send both the raw stream and a transformed derived stream into separate Eventhouse tables? When using Eventstream’s Event processing before ingestion mode, what are the trade-offs compared with direct ingestion followed by an Eventhouse update policy? How do teams handle changes to transformation logic when historical events need to be reprocessed? For reference-data enrichment, would you normally perform the lookup in Eventstream or after ingestion with KQL? Are five-minute or hourly aggregations better implemented in Eventstream, through an update policy, or with an Eventhouse materialized view? Microsoft’s Eventstream destination guidance documents both direct ingestion and event processing before ingestion, while the KQL update-policy documentation provides another way to transform ingested data. I would be interested to hear how others divide responsibility between Eventstream and Eventhouse in production, particularly where auditability, reprocessing and maintainability are important. Thanks in advance!Solved113Views0likes2CommentsStreaming Architecture
Hi everyone, I'm exploring Microsoft Fabric Real-Time Intelligence and would love to understand how it's being used in enterprise environments. Which business scenarios have benefited the most from real-time analytics? How do you balance: Low latency High throughput Cost Scalability Reliability I'm interested in learning from real production implementations. Thank you!Solved225Views0likes4CommentsFrom Rain Drops to Playability: Scoring Real-Time Weather Insights for FIFA World Cup 2026⚽
In this blog, I will walk you through how we can support the FIFA World Cup 2026 by organizing teams and players by delivering real-time, actionable weather insights. By predicting critical weather patterns - such as high humidity, extreme heat, and sudden rainfall - teams can proactively adjust their game-day strategies, and organizers can ensure optimal player safety. To bring this vision to life, we will build a comprehensive, real-time weather monitoring dashboard covering all 16 FIFA World Cup stadiums, powered entirely by the real-time analytics capabilities of Microsoft FabricEventstream connection with Event Hub
Hello, I have an existing event hub which I'm planning to use with an eventstream in Fabric. I see two connection options: 1. SAS keys 2. Organisational account I'm in a microsoft internal team and we're not allowed to use SAS keys. When I try to use my own account via organisational account option I'm getting following error Sign in failed with the following error code: "invalid_request". Error description given: "AADSTS65002: Consent between first party application 'b52893c8-bc2e-47fc-918b-77022b299bbc' and first party resource '80369ed6-5f11-4dd9-bef3-692475845e77' must be configured via preauthorization - applications owned and operated by Microsoft must get approval from the API owner before requesting tokens for that API. Trace ID: 3e344c2e-4ab0-442d-af25-23f3e1c15500 Correlation ID: 18583d2a-007b-4b7a-a7d0-5485087e55c7 Timestamp: 2025-07-29 16:05:32Z" (Session ID: f67ac96c-b700-4aed-8514-2b9acd5b00c8, Region: usfr) I'm wondering how to solve this and also is there any support to use Managed Identities for auth? This is normally possible but I don't see how via Microsoft Fabric.10KViews1like4CommentsOptions for High‑Frequency, Low‑Latency API Ingestion
I’m looking for guidance on options for invoking external APIs(with dynamic date parameters in specific format like '202602152359') at very high frequency (every 2 minutes or less) and making the data available with minimal end‑to‑end latency. Requirements: Invoke external APIs at 2‑minute intervals (or less). Parse incoming XML/JSON data into structured tables. Refresh user reports/streamlit dashboards quickly, alongside other enterprise data, to support near‑real‑time decision‑making wihtout the delay of sql endpoint refresh. We currently use a Fabric medallion architecture with payload based ingestions. However, this approach is not well‑suited for this use case, as running pipelines/notebooks on 2–5 minute schedules is triggering 70% Fabric capacity alerts.2.4KViews1like7Comments