Anomaly Detection

[DennesTorres]

Hi,

The walkthrough for real-time analysis in Fabric has the query below as an example of anomaly detection in taxi driver tips in New York:

nyctaxitrips

| lookup (Locations) on $left.PULocationID==$right.LocationID

| where Borough == "Manhattan"

| make-series s1 = avg(tip_amount) on tpep_pickup_datetime from datetime(2022-06-01) to datetime(2022-06-04) step 1h

| extend anomalies = series_decompose_anomalies(s1)

| render anomalychart with (anomalycolumns=anomalies)

What bothers me is that the detection is made over an average, so we are not really detecting the anomaly, we are detecting the hour inside which the anomaly happened.

The make-series only work with an aggregation function. Is it possible to use series_decompose_anomalies to detect individual anomaly values, instead of aggregated anomaly values? How could we use the make series without aggregation?

The example contained in the description page of the function series_decompose_anomalies is about web traffic and makes complete sense, because there is no anomaly on an individual value, it only exists on an aggregation. But for taxi driver tips, there is one anomaly and we can't find it aggregating.

Of course I could use separate queries over the hours pointed as anomaly to discover which is the anomaly value, but I was willing to find a solution to see the anomalies in a chart with a query.

Am I wrong? Is my concept wrong?

Kind Regards,

Dennes

[v-cboorla-msft]

Hi @DennesTorres 

Following up to see if the below suggestion was helpful. And, please do let us know in case of any further queries.

[v-cboorla-msft]

Hi @DennesTorres 

Welcome to Fabric Community and thanks for posting your question here.

As I understand that you are trying to use series_decompose_anomalies to detect individual anomaly values, instead of aggregated anomaly values. And you want to use the make-series without aggregation.

No, it is not possible to use the make-series operator without an aggregation function in MS Fabric KQL Query. The make-series operator is designed to create a series of aggregated values, and it requires an aggregation function to specify how the values should be aggregated.

For more information, please refer : make-series operator - Azure Data Explorer | Microsoft Learn

Yes, it is possible to use series_decompose_anomalies to detect individual anomaly values, instead of aggregated anomaly values in MS Fabric KQL. The function takes an expression containing a series (dynamic numerical array) as input, and extracts anomalous points with scores.

For more information, please refer : series_decompose_anomalies() - Azure Data Explorer | Microsoft Learn

I hope this information is helpful. Please let me know if you have any other questions.

[v-cboorla-msft]

Hi @DennesTorres 

Following up to see if the above suggestion was helpful. And, please do let us know in case of any further queries.