eventstream
11 TopicsFabric Business Events: what delivery guarantees and replay pattern should we design for?
Hi all, I am testing the newer Business Events capability in Fabric Real-Time Intelligence and trying to understand what reliability assumptions should be made for a production design. The pattern I am looking at is roughly: Eventstream → Business Event → Activator → downstream action / User Data Function with Eventhouse enabled so the published business events are also retained for historical analysis. The current documentation explains the publisher/consumer model and shows how Eventstream can publish a governed business event that Activator then consumes. What I have not been able to find clearly documented is the delivery contract between the published business event and its consumers. A few things I am trying to clarify: If an Activator consumer or downstream action is temporarily unavailable, does Fabric retry delivery of the business event? Should consumers assume at-least-once delivery and therefore be designed to handle duplicate events, or is a different delivery model used? Is event ordering guaranteed in any scope, for example for events from the same Eventstream publisher? Since published business events can also be retained automatically in Eventhouse, is that retained history intended to support replay/reprocessing after a consumer outage, or is it primarily an analytical record and replay would need to be implemented separately? Are there documented retry or delivery-retention windows that should be considered when designing an operational workflow? I am mainly trying to understand what a resilient production pattern should look like when the business event triggers something with side effects, where processing the same event twice or silently missing an event would matter. Would you generally make the downstream consumer idempotent and treat Eventhouse as an audit/recovery store, or is there a more Fabric-native pattern for this? Interested to hear how others are approaching this with Business Events and Activator.41Views0likes2CommentsBest practice for handling schema evolution in Fabric Eventstream before data reaches Eventhouse?
I have an Eventstream receiving operational events where the schema may evolve over time. For example, the producer initially sends: DeviceId, Timestamp, Temperature, Status but later adds fields such as: Location, FirmwareVersion, ErrorCode I want the pipeline to continue ingesting events without breaking downstream KQL tables, update policies, materialized views, or Real-Time Dashboards. I am trying to understand where schema evolution should ideally be handled in a production Fabric RTI architecture. Would you: enforce the contract upstream using Schema Registry normalize changing fields inside Eventstream before Eventhouse ingestion land the raw payload first and handle schema evolution inside Eventhouse/KQL maintain separate versioned event schemas/tables How are people handling this in production when producers can add fields without notice? I am particularly interested in avoiding a design where every small upstream schema change forces updates across Eventstream, KQL tables, update policies, and downstream dashboards.Solved89Views0likes2CommentsBest practice for deciding between Eventstream transformations and Eventhouse update policies
Hi Fabric Community, I am exploring a Real-Time Intelligence architecture and would appreciate some guidance on where transformation logic should ideally be placed. The proposed flow is: Azure Event Hubs → Fabric Eventstream → Eventhouse → Real-Time Dashboard / Power BI Fabric Eventstream supports filtering, field management, aggregation and other processing before events are written to the destination. An Eventhouse can also ingest the raw events first and transform them into curated tables through KQL update policies. I am trying to understand the recommended boundary between these two layers. For example, assume the incoming event contains: Device or customer identifier Event timestamp Event type Location Numeric readings Additional JSON properties The required processing includes: Removing events that fail basic validation Renaming and standardizing fields Converting timestamps and data types Flattening selected JSON properties Enriching the event with reference data Creating five-minute aggregates Preserving the original event for auditing and future reprocessing My current thinking is: Use Eventstream for lightweight filtering, routing and simple schema normalization. Land the original event in a Bronze table whenever replay or auditing is required. Use Eventhouse update policies or KQL for enrichment, reusable business logic and curated Silver tables. Use materialized views for frequently queried aggregations rather than calculating them repeatedly in dashboards. However, I am unsure where Microsoft recommends drawing the line. A few questions: Are there transformation types that should generally remain in Eventstream rather than Eventhouse? Is it considered good practice to send both the raw stream and a transformed derived stream into separate Eventhouse tables? When using Eventstream’s Event processing before ingestion mode, what are the trade-offs compared with direct ingestion followed by an Eventhouse update policy? How do teams handle changes to transformation logic when historical events need to be reprocessed? For reference-data enrichment, would you normally perform the lookup in Eventstream or after ingestion with KQL? Are five-minute or hourly aggregations better implemented in Eventstream, through an update policy, or with an Eventhouse materialized view? Microsoft’s Eventstream destination guidance documents both direct ingestion and event processing before ingestion, while the KQL update-policy documentation provides another way to transform ingested data. I would be interested to hear how others divide responsibility between Eventstream and Eventhouse in production, particularly where auditability, reprocessing and maintainability are important. Thanks in advance!Solved95Views0likes2CommentsStreaming Architecture
Hi everyone, I'm exploring Microsoft Fabric Real-Time Intelligence and would love to understand how it's being used in enterprise environments. Which business scenarios have benefited the most from real-time analytics? How do you balance: Low latency High throughput Cost Scalability Reliability I'm interested in learning from real production implementations. Thank you!Solved213Views0likes4CommentsDataverse Data (Preview) feature in Fabric RTI Error
I am trying to test the new Dataverse Data (Preview) feature for streaming data from Dataverse to Fabric. After using the preview option for source data from Dataverse, I can also choose a table and events such as Create, Update, Delete to track. However, when everyting is finished I get this error: " MessageFailed to create data source: dataverse due to the following error: E0179001: Cannot connect to Kafka. Please check your configuration. Details E0179001: Cannot connect to Kafka. Please check your configuration. " What is the resolution for this? Is this new preview feeature functional?883Views0likes6CommentsError in Microsoft Applied Skills: Implement a Real-Time Intelligence Assessment Lab : Need help
Hello All, since a week I was stuck on one error in Microsoft Applied Skills: Implement a Real-Time Intelligence solution with Microsoft Fabric Lab. Did anyone face the same issue if so, can anyone help me out how to resolve it. Attaching the error and related images below. Error: It says " unable to load container blobs. Verify that workspace identity is enabled and has storage blob data reader access to the storage account" . I am familiar with this permission and have enabled it in my own azure environment for my own blob that I have created but since these are pre created by Microsoft, I don't have access to these storage blobs and hence I am not sure how to proceed here. I have attempted this twice but I was stuck at the same point. I really appreciate your help and taking time in looking into my post. Thank you so much.Solved1.8KViews0likes7CommentsHow to reset Eventhouse Ingestion from Batching to Streaming ?
Hi everyone, I am experiencing a persistent latency issue in our Microsoft Fabric Eventhouse. A table that was previously performing perfectly has suddenly "downgraded" its ingestion path from Streaming to Batching and refuses to recover. The Situation: The Problem: In our PROD Eventhouse, ingestion latency is stuck at 12–15 seconds. The table is generating 45+ shards (extents) every 10 minutes, confirming it is in Batching Mode. The Discrepancy: Our DEV Eventhouse (identical schema and higher data volume) is still Streaming perfectly with ~5-second latency and 0 shards created per 10 minutes. The History: PROD was working fine (5s latency) and then switched to this slow Batching state on its own without any schema or policy changes. What we have verified: Streaming Policy: Both DEV and PROD have streamingingestion enabled. Batching Policy: We tested various ingestionbatching settings in a separate environment. We confirmed that the 12s latency in PROD is a result of the Batching Path overhead Unable to restore streaming: Running .alter table ... streamingingestion enable in PROD does not trigger a return to the Streaming. Our Questions: Why would an Eventhouse suddenly "blacklist" a table from the Streaming path and move it to permanent Batching if the Capacity (CU) is healthy (~50%)? How do we reverse this process? Once a table is stuck in this "Permanent Batching" state, what is the specific command or workflow to force the Eventhouse to re-evaluate it for the Streaming (Fast) Lane? We need to restore the 5-second latency to meet our real-time requirements. Any insights into the internal health-check logic of the Eventhouse would be incredibly helpful. Thank you !Solved32KViews0likes15CommentsArchitecture question and brainstorming
Hello Community, I have a design question that I need help with, we have a lot of vendor data that comes in form of CSV, XML, XLSX files via sftp. I already have ADF that copies this files into ADLS Gen2. Now when implementing in fabric I actually used blob storage events to trigger a notebook to load this files into a Lakehouse. I want to get suggestions if it's a good idea to use real time intelligence to stream data from this files and stage into a lakehouse? Has anyone implemented this? If so would love to get guidance. I am fairly new to evetstream and need more details on CU consumption, cost etc. what are disadvantages of using streaming notebook?Solved4.2KViews0likes4CommentsNot Able to create a Connection between Fabric EventStream and Azure Event Hub.
Working on a real-time data pipeline using 𝗠𝗶𝗰𝗿𝗼𝘀𝗼𝗳𝘁 𝗙𝗮𝗯𝗿𝗶𝗰 𝗘𝘃𝗲𝗻𝘁𝘀𝘁𝗿𝗲𝗮𝗺 and 𝗔𝘇𝘂𝗿𝗲 𝗘𝘃𝗲𝗻𝘁 𝗛𝘂𝗯, and I ran into an issue I’m hoping the community might have seen before. While trying to create a new Azure Event Hub connection inside Fabric Eventstream, the connection configuration panel appears completely 𝗯𝗹𝗮𝗻𝗸, so I’m unable to enter the required details like the connection string or namespace. 𝗧𝗵𝗶𝗻𝗴𝘀 𝗜’𝘃𝗲 𝗮𝗹𝗿𝗲𝗮𝗱𝘆 𝘁𝗿𝗶𝗲𝗱: • Refreshing the browser and clearing cache. • Using different browsers (Edge, Chrome). • Logging out and back into Fabric. Still facing the same issue. Has anyone encountered this while integrating Fabric Eventstream with Azure Event Hub? Any suggestions or configuration checks I should try? Appreciate any insights from the communitySolved2.6KViews0likes3CommentsAzure Maps base layer not loading in Real-Time Dashboard - Only points visible
Hi Fabric Community, I'm experiencing an issue with Map visualizations in Real-Time Dashboards. The data points are rendering correctly, but the Azure Maps base layer (streets, terrain, satellite) is not loading - only a white/blank background appears. **Environment:** - Microsoft Fabric Real-Time Intelligence - Real-Time Dashboard with KQL queries - Eventhouse as data source - Browser: Chrome/Edge (tested both) - Region: South Central US **Steps to reproduce:** 1. Create a KQL query with latitude/longitude columns 2. Add to Real-Time Dashboard 3. Change visual type to "Map" 4. Configure: Latitude column, Longitude column, Label column 5. Points appear but no map background **What I see:** - Colored circles representing my 10 GPS devices - Correct relative positions between points - Zoom controls (+/-) present but don't show map tiles - "Center" button works for positioning **What I expected:** - Azure Maps base layer with streets/roads visible behind the points **Troubleshooting attempted:** - Tried different browsers (Chrome, Edge) - Cleared browser cache - Verified coordinates are correct (Mexico City area: 19.5, -99.3) - Checked browser console - no obvious errors **Questions:** 1. Is there a known issue with Azure Maps in Real-Time Dashboards? 2. Are there specific tenant/admin settings required for Azure Maps? 3. Is this a preview limitation? Screenshots attached showing points without map background. Thank you for any guidance!Solved4.1KViews0likes2Comments