eventstream

163 Topics

Capacity overview events and Timepoints details
Good morning, is there any update on when the information about the operations provided by the 'Capacity overview events' connector from an eventstream will be included? thanks in advance
rgsalido
1 day ago Place Eventstream
115Views
0likes
7Comments
Fabric Business Events: what delivery guarantees and replay pattern should we design for?
Hi all, I am testing the newer Business Events capability in Fabric Real-Time Intelligence and trying to understand what reliability assumptions should be made for a production design. The pattern I am looking at is roughly: Eventstream → Business Event → Activator → downstream action / User Data Function with Eventhouse enabled so the published business events are also retained for historical analysis. The current documentation explains the publisher/consumer model and shows how Eventstream can publish a governed business event that Activator then consumes. What I have not been able to find clearly documented is the delivery contract between the published business event and its consumers. A few things I am trying to clarify: If an Activator consumer or downstream action is temporarily unavailable, does Fabric retry delivery of the business event? Should consumers assume at-least-once delivery and therefore be designed to handle duplicate events, or is a different delivery model used? Is event ordering guaranteed in any scope, for example for events from the same Eventstream publisher? Since published business events can also be retained automatically in Eventhouse, is that retained history intended to support replay/reprocessing after a consumer outage, or is it primarily an analytical record and replay would need to be implemented separately? Are there documented retry or delivery-retention windows that should be considered when designing an operational workflow? I am mainly trying to understand what a resilient production pattern should look like when the business event triggers something with side effects, where processing the same event twice or silently missing an event would matter. Would you generally make the downstream consumer idempotent and treat Eventhouse as an audit/recovery store, or is there a more Fabric-native pattern for this? Interested to hear how others are approaching this with Business Events and Activator.
ShivekMaharaj
7 days ago Place Real-Time Intelligence
53Views
0likes
2Comments
Best practice for handling schema evolution in Fabric Eventstream before data reaches Eventhouse?
I have an Eventstream receiving operational events where the schema may evolve over time. For example, the producer initially sends: DeviceId, Timestamp, Temperature, Status but later adds fields such as: Location, FirmwareVersion, ErrorCode I want the pipeline to continue ingesting events without breaking downstream KQL tables, update policies, materialized views, or Real-Time Dashboards. I am trying to understand where schema evolution should ideally be handled in a production Fabric RTI architecture. Would you: enforce the contract upstream using Schema Registry normalize changing fields inside Eventstream before Eventhouse ingestion land the raw payload first and handle schema evolution inside Eventhouse/KQL maintain separate versioned event schemas/tables How are people handling this in production when producers can add fields without notice? I am particularly interested in avoiding a design where every small upstream schema change forces updates across Eventstream, KQL tables, update policies, and downstream dashboards.
Solved
ShivekMaharaj
12 days ago Place Real-Time Intelligence
96Views
0likes
2Comments
Best practice for deciding between Eventstream transformations and Eventhouse update policies
Hi Fabric Community, I am exploring a Real-Time Intelligence architecture and would appreciate some guidance on where transformation logic should ideally be placed. The proposed flow is: Azure Event Hubs → Fabric Eventstream → Eventhouse → Real-Time Dashboard / Power BI Fabric Eventstream supports filtering, field management, aggregation and other processing before events are written to the destination. An Eventhouse can also ingest the raw events first and transform them into curated tables through KQL update policies. I am trying to understand the recommended boundary between these two layers. For example, assume the incoming event contains: Device or customer identifier Event timestamp Event type Location Numeric readings Additional JSON properties The required processing includes: Removing events that fail basic validation Renaming and standardizing fields Converting timestamps and data types Flattening selected JSON properties Enriching the event with reference data Creating five-minute aggregates Preserving the original event for auditing and future reprocessing My current thinking is: Use Eventstream for lightweight filtering, routing and simple schema normalization. Land the original event in a Bronze table whenever replay or auditing is required. Use Eventhouse update policies or KQL for enrichment, reusable business logic and curated Silver tables. Use materialized views for frequently queried aggregations rather than calculating them repeatedly in dashboards. However, I am unsure where Microsoft recommends drawing the line. A few questions: Are there transformation types that should generally remain in Eventstream rather than Eventhouse? Is it considered good practice to send both the raw stream and a transformed derived stream into separate Eventhouse tables? When using Eventstream’s Event processing before ingestion mode, what are the trade-offs compared with direct ingestion followed by an Eventhouse update policy? How do teams handle changes to transformation logic when historical events need to be reprocessed? For reference-data enrichment, would you normally perform the lookup in Eventstream or after ingestion with KQL? Are five-minute or hourly aggregations better implemented in Eventstream, through an update policy, or with an Eventhouse materialized view? Microsoft’s Eventstream destination guidance documents both direct ingestion and event processing before ingestion, while the KQL update-policy documentation provides another way to transform ingested data. I would be interested to hear how others divide responsibility between Eventstream and Eventhouse in production, particularly where auditability, reprocessing and maintainability are important. Thanks in advance!
Solved
ShivekMaharaj
19 days ago Place Real-Time Intelligence
96Views
0likes
2Comments
Streaming Architecture
Hi everyone, I'm exploring Microsoft Fabric Real-Time Intelligence and would love to understand how it's being used in enterprise environments. Which business scenarios have benefited the most from real-time analytics? How do you balance: Low latency High throughput Cost Scalability Reliability I'm interested in learning from real production implementations. Thank you!
Solved
binitafulpagare
1 month ago Place Real-Time Intelligence
214Views
0likes
4Comments
Eventstream connection with Event Hub
Hello, I have an existing event hub which I'm planning to use with an eventstream in Fabric. I see two connection options: 1. SAS keys 2. Organisational account I'm in a microsoft internal team and we're not allowed to use SAS keys. When I try to use my own account via organisational account option I'm getting following error Sign in failed with the following error code: "invalid_request". Error description given: "AADSTS65002: Consent between first party application 'b52893c8-bc2e-47fc-918b-77022b299bbc' and first party resource '80369ed6-5f11-4dd9-bef3-692475845e77' must be configured via preauthorization - applications owned and operated by Microsoft must get approval from the API owner before requesting tokens for that API. Trace ID: 3e344c2e-4ab0-442d-af25-23f3e1c15500 Correlation ID: 18583d2a-007b-4b7a-a7d0-5485087e55c7 Timestamp: 2025-07-29 16:05:32Z" (Session ID: f67ac96c-b700-4aed-8514-2b9acd5b00c8, Region: usfr) I'm wondering how to solve this and also is there any support to use Managed Identities for auth? This is normally possible but I don't see how via Microsoft Fabric.
iremugurlu
1 month ago Place Eventstream
10KViews
1like
4Comments
Options for High‑Frequency, Low‑Latency API Ingestion
I’m looking for guidance on options for invoking external APIs(with dynamic date parameters in specific format like '202602152359') at very high frequency (every 2 minutes or less) and making the data available with minimal end‑to‑end latency. Requirements: Invoke external APIs at 2‑minute intervals (or less). Parse incoming XML/JSON data into structured tables. Refresh user reports/streamlit dashboards quickly, alongside other enterprise data, to support near‑real‑time decision‑making wihtout the delay of sql endpoint refresh. We currently use a Fabric medallion architecture with payload based ingestions. However, this approach is not well‑suited for this use case, as running pipelines/notebooks on 2–5 minute schedules is triggering 70% Fabric capacity alerts.
prathijp
1 month ago Place Eventstream
2.4KViews
1like
7Comments
Heartbeat Action Query Not Sent to Source
I created an Eventstream with Postgre SQL CDC as source. I want to use heartbeat action query to ensure that Eventstream to keep the replication slot size low in source. There is no issue/error in the Eventstream but the heartbeat query is never been sent to source. Is there a configuration I missed? This is the table that I created and the initial value: CREATE TABLE public.debezium_heartbeat ( id SERIAL PRIMARY KEY, timestamp TIMESTAMP WITH TIME ZONE ); INSERT INTO public.debezium_heartbeat (id, timestamp) VALUES (1, now()); This is the heartbeat action query that I put on the Eventstream source: UPDATE public.debezium_heartbeat SET timestamp = now() WHERE id = 1
Solved
lyancy
1 month ago Place Eventstream
1.2KViews
1like
7Comments
Disable multiple eventstreams
Hello Is there a way to disable multiple evenstreams in one go rather than having to go into each one and deactivate manually? I'm basically looking to get some code to run if at all possible. Thanks
Solved
RichCross
1 month ago Place Eventstream
523Views
0likes
2Comments
Dataverse Data (Preview) feature in Fabric RTI Error
I am trying to test the new Dataverse Data (Preview) feature for streaming data from Dataverse to Fabric. After using the preview option for source data from Dataverse, I can also choose a table and events such as Create, Update, Delete to track. However, when everyting is finished I get this error: " MessageFailed to create data source: dataverse due to the following error: E0179001: Cannot connect to Kafka. Please check your configuration. Details E0179001: Cannot connect to Kafka. Please check your configuration. " What is the resolution for this? Is this new preview feeature functional?
Aavanade_365
2 months ago Place Real-Time Intelligence
888Views
0likes
6Comments