general question
29 TopicsFabric Business Events: what delivery guarantees and replay pattern should we design for?
Hi all, I am testing the newer Business Events capability in Fabric Real-Time Intelligence and trying to understand what reliability assumptions should be made for a production design. The pattern I am looking at is roughly: Eventstream → Business Event → Activator → downstream action / User Data Function with Eventhouse enabled so the published business events are also retained for historical analysis. The current documentation explains the publisher/consumer model and shows how Eventstream can publish a governed business event that Activator then consumes. What I have not been able to find clearly documented is the delivery contract between the published business event and its consumers. A few things I am trying to clarify: If an Activator consumer or downstream action is temporarily unavailable, does Fabric retry delivery of the business event? Should consumers assume at-least-once delivery and therefore be designed to handle duplicate events, or is a different delivery model used? Is event ordering guaranteed in any scope, for example for events from the same Eventstream publisher? Since published business events can also be retained automatically in Eventhouse, is that retained history intended to support replay/reprocessing after a consumer outage, or is it primarily an analytical record and replay would need to be implemented separately? Are there documented retry or delivery-retention windows that should be considered when designing an operational workflow? I am mainly trying to understand what a resilient production pattern should look like when the business event triggers something with side effects, where processing the same event twice or silently missing an event would matter. Would you generally make the downstream consumer idempotent and treat Eventhouse as an audit/recovery store, or is there a more Fabric-native pattern for this? Interested to hear how others are approaching this with Business Events and Activator.1View0likes0CommentsBest practice for handling schema evolution in Fabric Eventstream before data reaches Eventhouse?
I have an Eventstream receiving operational events where the schema may evolve over time. For example, the producer initially sends: DeviceId, Timestamp, Temperature, Status but later adds fields such as: Location, FirmwareVersion, ErrorCode I want the pipeline to continue ingesting events without breaking downstream KQL tables, update policies, materialized views, or Real-Time Dashboards. I am trying to understand where schema evolution should ideally be handled in a production Fabric RTI architecture. Would you: enforce the contract upstream using Schema Registry normalize changing fields inside Eventstream before Eventhouse ingestion land the raw payload first and handle schema evolution inside Eventhouse/KQL maintain separate versioned event schemas/tables How are people handling this in production when producers can add fields without notice? I am particularly interested in avoiding a design where every small upstream schema change forces updates across Eventstream, KQL tables, update policies, and downstream dashboards.Solved82Views0likes2CommentsBest practice for deciding between Eventstream transformations and Eventhouse update policies
Hi Fabric Community, I am exploring a Real-Time Intelligence architecture and would appreciate some guidance on where transformation logic should ideally be placed. The proposed flow is: Azure Event Hubs → Fabric Eventstream → Eventhouse → Real-Time Dashboard / Power BI Fabric Eventstream supports filtering, field management, aggregation and other processing before events are written to the destination. An Eventhouse can also ingest the raw events first and transform them into curated tables through KQL update policies. I am trying to understand the recommended boundary between these two layers. For example, assume the incoming event contains: Device or customer identifier Event timestamp Event type Location Numeric readings Additional JSON properties The required processing includes: Removing events that fail basic validation Renaming and standardizing fields Converting timestamps and data types Flattening selected JSON properties Enriching the event with reference data Creating five-minute aggregates Preserving the original event for auditing and future reprocessing My current thinking is: Use Eventstream for lightweight filtering, routing and simple schema normalization. Land the original event in a Bronze table whenever replay or auditing is required. Use Eventhouse update policies or KQL for enrichment, reusable business logic and curated Silver tables. Use materialized views for frequently queried aggregations rather than calculating them repeatedly in dashboards. However, I am unsure where Microsoft recommends drawing the line. A few questions: Are there transformation types that should generally remain in Eventstream rather than Eventhouse? Is it considered good practice to send both the raw stream and a transformed derived stream into separate Eventhouse tables? When using Eventstream’s Event processing before ingestion mode, what are the trade-offs compared with direct ingestion followed by an Eventhouse update policy? How do teams handle changes to transformation logic when historical events need to be reprocessed? For reference-data enrichment, would you normally perform the lookup in Eventstream or after ingestion with KQL? Are five-minute or hourly aggregations better implemented in Eventstream, through an update policy, or with an Eventhouse materialized view? Microsoft’s Eventstream destination guidance documents both direct ingestion and event processing before ingestion, while the KQL update-policy documentation provides another way to transform ingested data. I would be interested to hear how others divide responsibility between Eventstream and Eventhouse in production, particularly where auditability, reprocessing and maintainability are important. Thanks in advance!Solved90Views0likes2CommentsICS (Industial Controls Systems)/PLC (Programmable Logic Controller) Integration
Can Microsoft Fabric be integrated with ICS (Industrial Controls Systems) including PLC (Programmable Logic Controller) and HMI (Human-Machine Interface)? We're interested in exploring Data Activator with our ICS information. Thank you for your time!2.8KViews1like3CommentsError in Microsoft Applied Skills: Implement a Real-Time Intelligence Assessment Lab : Need help
Hello All, since a week I was stuck on one error in Microsoft Applied Skills: Implement a Real-Time Intelligence solution with Microsoft Fabric Lab. Did anyone face the same issue if so, can anyone help me out how to resolve it. Attaching the error and related images below. Error: It says " unable to load container blobs. Verify that workspace identity is enabled and has storage blob data reader access to the storage account" . I am familiar with this permission and have enabled it in my own azure environment for my own blob that I have created but since these are pre created by Microsoft, I don't have access to these storage blobs and hence I am not sure how to proceed here. I have attempted this twice but I was stuck at the same point. I really appreciate your help and taking time in looking into my post. Thank you so much.Solved1.8KViews0likes7CommentsRealtime - Business Events - cannot create business event shcema
Hi Has anyone succeeded in creating business event shcema? I keep getting the error "An unexpected error occourred". I have tried waiting a week but it seems we simply cannot make a "business event schema" (se screendump in appendix). Our tennant is in West Europe. Appendix - screendumpsSolved1.1KViews0likes1CommentArchitecture question and brainstorming
Hello Community, I have a design question that I need help with, we have a lot of vendor data that comes in form of CSV, XML, XLSX files via sftp. I already have ADF that copies this files into ADLS Gen2. Now when implementing in fabric I actually used blob storage events to trigger a notebook to load this files into a Lakehouse. I want to get suggestions if it's a good idea to use real time intelligence to stream data from this files and stage into a lakehouse? Has anyone implemented this? If so would love to get guidance. I am fairly new to evetstream and need more details on CU consumption, cost etc. what are disadvantages of using streaming notebook?Solved4.2KViews0likes4CommentsArchitecture & GPS hardware
Hello Fabric community, I'm building a fleet monitoring platform for trucks/vehicles and need architectural guidance for using Microsoft Fabric Real-Time Intelligence, as well as GPS hardware recommendations. MY SCENARIO: ------------ - GPS trackers sending location data every 10-60 seconds via TCP - Data includes: GPS coordinates, speed, fuel level, engine RPM, temperature sensors, driver behavior (harsh braking, acceleration) - Need to support multiple tenants (each customer sees only their own fleet) - Planning to scale from 50 vehicles to 1000+ vehicles - Want real-time dashboards + historical analytics + automated alerts - Future: add video/dashcam capabilities MY PLANNED ARCHITECTURE: ------------------------ GPS Devices → TCP Server (Traccar/custom) → Azure Event Hub → Fabric Eventstream → Eventhouse → Real-Time Dashboard QUESTIONS: GPS Hardware: ------------- 1. What GPS tracker brands/models do you recommend that integrate well with custom backends and Microsoft Fabric? I'm considering Teltonika FMC920, Queclink, or Concox - any experience with these? 2. Which GPS devices have the best open protocols for parsing data and sending to Azure/Fabric? 3. Any recommendations for GPS trackers with OBD-II support to read vehicle diagnostics (fuel, RPM, error codes)? Microsoft Fabric Architecture: ------------------------------ 4. Is Eventstream + Eventhouse the right approach for high-frequency GPS telemetry data? 5. What's the recommended way to implement multi-tenancy (row-level security) for different fleet customers? 6. Should I use KQL Database or Lakehouse for storing historical GPS data (months/years)? 7. Any recommendations for handling geofencing alerts in real-time using Activator? 8. What Fabric SKU (F4, F8, F16?) would you recommend starting with for ~100 vehicles sending data every 30 seconds? I'm based in Mexico and planning to deploy this commercially for fleet owners. Any guidance from people who have built similar IoT/telematics solutions with Fabric would be greatly appreciated! Thanks in advance!Solved6.7KViews0likes4CommentsRTI Event Metadata Table Use Cases
Hi everyone, I’m looking to expand how others are using metadata tables in Microsoft Fabric RTI architectures. Current state We have an IaC-driven deployment process that provisions RTI assets from source → bronze → silver. As part of this deployment, we persist configuration and lineage information into a metadata table stored in SQL DB. At a high level, the metadata schema includes: EventTopic SourceObjectSettings (JSON) – bronze/silver table names, flatten function, etc. TopicInformation (JSON) – topic-level semantics Audit columns (create/modify user + timestamps) The metadata is written at deploy time and represents the intended RTI topology. Current primary use case: Bronze → Silver reprocessing Our main use case today is controlled reprocessing from bronze to silver. Since Fabric update policies only apply to data in motion, when we need to: fix a flattening bug handle schema changes correct downstream logic we need a way to deterministically replay historical data. To support this, I built a parameterized pipeline that accepts: EventTopic StartTime EndTime The pipeline triggers a fully parameterized notebook that: Uses the provided EventTopic to query the metadata table Retrieves the associated flatten function, bronze table, and silver table Dynamically builds the KQL query Reprocesses bronze data for the specified time window into silver This approach avoids hard-coded topic logic and has worked well so far. What I’m looking for I’m interested in learning how others are extending this metadata-driven pattern beyond reprocessing, particularly in larger RTI platforms. What other valuable use cases have you found for metadata tables in RTI architectures? Any lessons learned or pitfalls would also be appreciated. Thanks in advance.Solved7.1KViews2likes4CommentsMonitor all the pipeline failures and send a Teams alert
We run multiple scheduled pipelines, each with chained Invoke pipelines. How can we monitor all of the multiple pipeline failures and send a Teams alert when any pipeline fails? Please give step-by-step instructions.Solved4.5KViews0likes4Comments