real-time analytics
35 TopicsNo Start/Stop on event stream ?
Hi, I built an eventstream with two targets. The problem is: Once I completed the configuration of the first target, the ingestion started, leaving the 2nd target behind. Should the eventstream have a start/stop button to ensure both targets will be in sync? Am I missing something, or is this a feature request? Kind Regards, Dennes27KViews0likes10CommentsIngest Data from Streaming API
I am trying to ingest data into my Lakehouse from a streaming API (https://developer-specs.company-information.service.gov.uk/streaming-api/guides/overview) I currently do this using a Function App that connects to the API and packages and stores files every 8 minutes. The Function then reconnects using the latest timestamp processed. This is the recommended method in the above link. How can I recreate this in Fabric pipelines/eventstream to ingest the streaming data into OneLake? ThanksSolved20KViews0likes6CommentsHigh latency in Fabric Eventstream using event processing with KQL Database
Hello! I am testing Fabric real-time analytics with KQL databases and eventstreaming from Event Hub. Using Fabric Eventstream with direct ingestion to a KQL database, I get the latency I expect (a couple of seconds) and it seems to work fine. However when I add event processing before ingestion, with a simple filter operation and mapping columns operation, the ingestion latency into the KQL database seems to go up to 30 seconds. In my test I have about 20 small events being sent every second. The watermark delay graph in the EventStream still shows 3 second, but as I said the latency is higher. The inserts into the KQL database seems to get batched together in larger groups and inserted with much higher delay than when using Eventstream with direct ingestion. I am thinking it maybe uses Kusto Ingestion batching, instead of Streaming Ingestion? Is this something that is expected, and is there a way to alter settings to reduce the delay? Best regards, EmilSolved5.7KViews1like2CommentsHow to feed Streaming data to Custom App in Eventstream in Realtime Analytics Fabric using Fabric
Hi, I want to stream data stored in a csv file in Azure Storage account by feeding data to Custom App Event Hubs data source in Eventstream in Real time Analytics in Fabric. I want to use a Fabric solution to implement this. Any help on this would be appreciated. Thanks.3KViews0likes2CommentsWhere is Sample data coming from for using Sample data as a streaming source in Fabric eventstream
Hi, We want to mimic how the Sample data as a source in Eventstream under Real time Anlytics in Fabric is pulling the Sample data from (like: Sample db placed in some cloud platform, github, etc.) for it to allow streaming data. Basically, we want to understand what is the source for Sample data source in eventstream and how is it able to have streaming data behaviour. We want to replicate this behaviour for implementation of our Realtime analytics use case which needs a source for streaming the data and we want some other sources except Azure IOT hub and Azure Event hub. Please help with any links on this or any information. Thanks.Solved6.5KViews0likes1CommentWhat is KQL db in fabric generally used for storing, is it historical data or only streaming data?
Hi, I wanted to understand if usually end users utilize KQL db in fabric for storing historical data as well or only streaming data. What is usual retention period commonly followed for KQL db as the default is unlimited. Thanks in advance.3.2KViews0likes1CommentAnomaly Detection
Hi, The walkthrough for real-time analysis in Fabric has the query below as an example of anomaly detection in taxi driver tips in New York: nyctaxitrips | lookup (Locations) on $left.PULocationID==$right.LocationID | where Borough == "Manhattan" | make-series s1 = avg(tip_amount) on tpep_pickup_datetime from datetime(2022-06-01) to datetime(2022-06-04) step 1h | extend anomalies = series_decompose_anomalies(s1) | render anomalychart with (anomalycolumns=anomalies) What bothers me is that the detection is made over an average, so we are not really detecting the anomaly, we are detecting the hour inside which the anomaly happened. The make-series only work with an aggregation function. Is it possible to use series_decompose_anomalies to detect individual anomaly values, instead of aggregated anomaly values? How could we use the make series without aggregation? The example contained in the description page of the function series_decompose_anomalies is about web traffic and makes complete sense, because there is no anomaly on an individual value, it only exists on an aggregation. But for taxi driver tips, there is one anomaly and we can't find it aggregating. Of course I could use separate queries over the hours pointed as anomaly to discover which is the anomaly value, but I was willing to find a solution to see the anomalies in a chart with a query. Am I wrong? Is my concept wrong? Kind Regards, Dennes2.7KViews0likes3Comments