eventhouse
60 TopicsAnomaly Detector is disabled
Hello, I have a big Table in which we have our TimeSeries data from various devices. Since not every device sends the same data we have a big schema of numeric columns that are not filled for every deviceId. Is that the reason why there is no Anomaly Detector in my EventHouse? In the schema I can see that columns for the data are int or real. Some columns are string, but that shoudn't be the reason, right? Has anyone faced the same issue?11Views0likes2CommentsEventhouse Capacity Planner minimum CU not reflected in UI/API
Hi all, We've set a minimum of 32 CU on our Eventhouse via Capacity Planner (autoscale alone isn't sufficient - we need guaranteed baseline capacity to protect a large bulk-ingestion workload from destination-side OutOfMemory during a migration). We've been told the 32 CU setting has been applied on the backend, but the Fabric UI and REST API (.show cluster / .show diagnostics) still report capacity/behavior consistent with a much lower tier. Can anyone confirm: Whether the a custom minimum CU value is actually enforced server-side even though the UI/API don't show it, and When UI/API reporting is expected to catch up to reflect the configured minimum? Any insight or similar experience would be appreciated.82Views0likes3CommentsFabric Business Events: what delivery guarantees and replay pattern should we design for?
Hi all, I am testing the newer Business Events capability in Fabric Real-Time Intelligence and trying to understand what reliability assumptions should be made for a production design. The pattern I am looking at is roughly: Eventstream → Business Event → Activator → downstream action / User Data Function with Eventhouse enabled so the published business events are also retained for historical analysis. The current documentation explains the publisher/consumer model and shows how Eventstream can publish a governed business event that Activator then consumes. What I have not been able to find clearly documented is the delivery contract between the published business event and its consumers. A few things I am trying to clarify: If an Activator consumer or downstream action is temporarily unavailable, does Fabric retry delivery of the business event? Should consumers assume at-least-once delivery and therefore be designed to handle duplicate events, or is a different delivery model used? Is event ordering guaranteed in any scope, for example for events from the same Eventstream publisher? Since published business events can also be retained automatically in Eventhouse, is that retained history intended to support replay/reprocessing after a consumer outage, or is it primarily an analytical record and replay would need to be implemented separately? Are there documented retry or delivery-retention windows that should be considered when designing an operational workflow? I am mainly trying to understand what a resilient production pattern should look like when the business event triggers something with side effects, where processing the same event twice or silently missing an event would matter. Would you generally make the downstream consumer idempotent and treat Eventhouse as an audit/recovery store, or is there a more Fabric-native pattern for this? Interested to hear how others are approaching this with Business Events and Activator.51Views0likes2CommentsBest practice for handling schema evolution in Fabric Eventstream before data reaches Eventhouse?
I have an Eventstream receiving operational events where the schema may evolve over time. For example, the producer initially sends: DeviceId, Timestamp, Temperature, Status but later adds fields such as: Location, FirmwareVersion, ErrorCode I want the pipeline to continue ingesting events without breaking downstream KQL tables, update policies, materialized views, or Real-Time Dashboards. I am trying to understand where schema evolution should ideally be handled in a production Fabric RTI architecture. Would you: enforce the contract upstream using Schema Registry normalize changing fields inside Eventstream before Eventhouse ingestion land the raw payload first and handle schema evolution inside Eventhouse/KQL maintain separate versioned event schemas/tables How are people handling this in production when producers can add fields without notice? I am particularly interested in avoiding a design where every small upstream schema change forces updates across Eventstream, KQL tables, update policies, and downstream dashboards.Solved94Views0likes2CommentsBest practice for deciding between Eventstream transformations and Eventhouse update policies
Hi Fabric Community, I am exploring a Real-Time Intelligence architecture and would appreciate some guidance on where transformation logic should ideally be placed. The proposed flow is: Azure Event Hubs → Fabric Eventstream → Eventhouse → Real-Time Dashboard / Power BI Fabric Eventstream supports filtering, field management, aggregation and other processing before events are written to the destination. An Eventhouse can also ingest the raw events first and transform them into curated tables through KQL update policies. I am trying to understand the recommended boundary between these two layers. For example, assume the incoming event contains: Device or customer identifier Event timestamp Event type Location Numeric readings Additional JSON properties The required processing includes: Removing events that fail basic validation Renaming and standardizing fields Converting timestamps and data types Flattening selected JSON properties Enriching the event with reference data Creating five-minute aggregates Preserving the original event for auditing and future reprocessing My current thinking is: Use Eventstream for lightweight filtering, routing and simple schema normalization. Land the original event in a Bronze table whenever replay or auditing is required. Use Eventhouse update policies or KQL for enrichment, reusable business logic and curated Silver tables. Use materialized views for frequently queried aggregations rather than calculating them repeatedly in dashboards. However, I am unsure where Microsoft recommends drawing the line. A few questions: Are there transformation types that should generally remain in Eventstream rather than Eventhouse? Is it considered good practice to send both the raw stream and a transformed derived stream into separate Eventhouse tables? When using Eventstream’s Event processing before ingestion mode, what are the trade-offs compared with direct ingestion followed by an Eventhouse update policy? How do teams handle changes to transformation logic when historical events need to be reprocessed? For reference-data enrichment, would you normally perform the lookup in Eventstream or after ingestion with KQL? Are five-minute or hourly aggregations better implemented in Eventstream, through an update policy, or with an Eventhouse materialized view? Microsoft’s Eventstream destination guidance documents both direct ingestion and event processing before ingestion, while the KQL update-policy documentation provides another way to transform ingested data. I would be interested to hear how others divide responsibility between Eventstream and Eventhouse in production, particularly where auditability, reprocessing and maintainability are important. Thanks in advance!Solved96Views0likes2CommentsFabric Eventhouse: Capacity policy for .ingest inline is hard-capped at 1 concurrent operation
Problem When using the KQL Activity in Fabric Data Pipelines to execute .ingest inline commands against an Eventhouse KQL database, the effective ingestion capacity is always 1 concurrent operation — regardless of the Fabric capacity SKU (tested on F2 and F8) and regardless of the cluster capacity policy setting. Running .alter-merge cluster policy capacity with ClusterMaximumConcurrentOperations: 16 succeeds without error, but .show capacity still reports an effective ingestion capacity of 1. Any pipeline pattern that executes .ingest inline within a parallel ForEach immediately hits 429 TooManyRequests / ControlCommandThrottledException errors. Highlighting that the capacity is 1. This makes the KQL Activity (KustoQueryLanguage type) unusable for parallel ingestion scenarios — which is an important use case in data pipelines (logging, event emission, watermark updates during parallel table processing). Question / Request Do you recognize this issue? if so I have the below request: Increase the effective ingestion capacity for Fabric Eventhouse to match what the capacity policy allows (respect ClusterMaximumConcurrentOperations)Solved467Views0likes11CommentsCopilot for EventHouse Queryset
Hi, RTI has introduced the preview feature of Copilot for the Eventhouse Queryset, but is there any way to automate the Copilot usage, via API endpoint or SDKs? The idea is to programmatically generate relevant and meaningful Kusto queries for business insights, from the tables and datasets present in the KQL DB in Eventhouse using Copilot. Thanks in advance!6.7KViews1like7CommentsHow do you decide when to use Real-Time Intelligence instead of batch processing in Microsoft Fabric
Hi everyone, I'm learning Microsoft Fabric and recently started exploring Real-Time Intelligence. I understand that it can process streaming data, but I'm trying to understand when it's the right choice compared to traditional batch processing. I have a few questions: What types of business scenarios benefit the most from Real-Time Intelligence? When would you choose streaming over scheduled batch processing? What are some common real-world use cases you've worked on? Are there any performance or cost considerations that beginners should be aware of? I'd really appreciate hearing about your experiences and any best practices you recommend. Thank you!Solved336Views1like6CommentsDelay in Onelake Shortcut from Eventhouse to Lakehouse
I have a table in a KQL Database in an eventhouse that is hooked up to an Azure Eventhub. I have turned on One Lake storage for this table and added a shortcut to the lakehouse for this table. The table receives around 10-15k events per day and I have set the retention policy for the table to 30 days. The eventhouse table data was updating and getting reflected in the shortcut table within a few minutes when I first set it up. But now it takes almost an hour or more to see the latest event data in the shortcut table in my lakehouse. I turned the one lake availability off and then on again, and also tried updating the table policies but nothing seems to workSolved256Views1like4CommentsReal‑Time Dashboards cannot be created
Hi all. We’re currently facing an issue in the a Fabric tenant that prevents us from creating Real‑Time Dashboards. According to Microsoft documentation, this feature requires either a Fabric Premium capacity or a Premium Per User (PPU) license. Reference: Create a Real-Time Dashboard - Microsoft Fabric | Microsoft Learn For example: In one tenant (F4 capacity, North Europe) with a user licensed as Premium Per User, the feature works as expected. In another tenant (F16 capacity, Southeast Asia) with a user licensed as Pro, Real‑Time Dashboards cannot be created. The error shown is: “Real‑Time Dashboard item can only be created if this action is enabled in the tenant. Ask your tenant admin to enable this.” This error is confusing, as the option to enable the feature is no longer available in the Admin Portal. Anyone had a similar issue?6.5KViews2likes15Comments