data engineering
6 TopicsAnomaly Detection
Hi, The walkthrough for real-time analysis in Fabric has the query below as an example of anomaly detection in taxi driver tips in New York: nyctaxitrips | lookup (Locations) on $left.PULocationID==$right.LocationID | where Borough == "Manhattan" | make-series s1 = avg(tip_amount) on tpep_pickup_datetime from datetime(2022-06-01) to datetime(2022-06-04) step 1h | extend anomalies = series_decompose_anomalies(s1) | render anomalychart with (anomalycolumns=anomalies) What bothers me is that the detection is made over an average, so we are not really detecting the anomaly, we are detecting the hour inside which the anomaly happened. The make-series only work with an aggregation function. Is it possible to use series_decompose_anomalies to detect individual anomaly values, instead of aggregated anomaly values? How could we use the make series without aggregation? The example contained in the description page of the function series_decompose_anomalies is about web traffic and makes complete sense, because there is no anomaly on an individual value, it only exists on an aggregation. But for taxi driver tips, there is one anomaly and we can't find it aggregating. Of course I could use separate queries over the hours pointed as anomaly to discover which is the anomaly value, but I was willing to find a solution to see the anomalies in a chart with a query. Am I wrong? Is my concept wrong? Kind Regards, Dennes2.7KViews0likes3CommentsWhat File Types are supported in KQL DB ?
Hey community, I have a folder like this: year/month/Jan/ In Jan, I have sub folders like day 1, day 2, .... In these days folders, I have many files for instance day 1 has .json, .csv, and day 2 has .xml, .xlxs All of them have the same schema. So I have used a KQL to get them via pipeline and all of these different types of files were appended into a single table. I'd like to know what kinds of file formats it won't append, or not supported in a KQL DB. Thanks !Solved2KViews0likes1CommentDashboard Tile created from Kusto source doesn't refresh
Hi, The problem is exactly what the subject mentions. I think I may be hitting some limitation, something not suppported, but I need some clue about what it is to work around. The table from the semantic model uses direct query to the KQL database. The model is composed, but I ensured this table has no relationship with any other table using different storage models. What is causing this limitation (no refresh on tiles in a dashboard) and how to work around it? P.S: I'm aware report changes don't reflect on tiles, but that's not the problem. The test was done with no report change in the middle. Kind Regards, DennesSolved2.6KViews0likes4CommentsKQL Shortcut OneLake Option Missing
Hi there - We want to process data from Azure Data Explorer (Kusto). using Spark. I have created shortcut using this Create a database shortcut - Microsoft Fabric | Microsoft Learn . However, I don't see option for creating shorcut on top of KQL Shortcut database i.e I want to access this data under Lakehouse using externaltable option in order to process using spark. Created ShortCut using Azure Data Explorer (Kusto). Create a database shortcut - Microsoft Fabric | Microsoft Learn OneLakeAvailability Option: I can see OneLake Availablity Option for new kql database created in Fabric but it's missig in Above Screenshot Please suggest how to resolve the issue. Thanks, AbhiramSolved3.2KViews0likes4CommentsFabric - Real Time Analytics - Cannot create connection to external event hub
Hi, I am having trouble when adding an external event hub (from a different tenant). The problem is occuring when trying to create the connection to the event hub. I am using the name of the event hub namspace, and not the entire domain of the event hub namespace I have tried using the SAS credentials on both the event hub namespace and the event hub instance. The SAS keys have Manage permission. I get the following error message when I try to connect using both the SAS credentials for the event hub instance and the event hub namespace After I have created the connection (even though it failed to create as mentioned above, it still occurs in the list of connections for the event hub source), it gets this error. The SAS key have Manage permission. Would gladly appreciate if anyone can assist me. Thanks! -BjarteSolved3.7KViews0likes6Comments