Anomaly Detection

the anomaly detection processor uses the three sigma rule and allows you to define the number of standard deviations considered normal for a data set if the data is within the designated standard deviations of the mean, it is not considered to be anomalous anomaly detection overview anomaly detection by 3 sigma rule is a conventional heuristic used for an approximately normalized data set a rolling window of values are "observed", and their average and standard deviations are calculated you can define the number of standard deviations that would be considered "normal", hence everything beyond that would be anomalous data in case of an anomaly, the standard deviation window is expanded ever so slightly, so that it can adjust if this anomalous data becomes seasonal this calculation is done because in live data, it is sometimes not preferable for one large anomaly to drastically change the moving average and moving standard deviations with the control chart mode , this calculation can be completely bypassed, if all you need to check is whether the value is within upper/lower limit or not if control chart mode is enabled, there will be no modifications done to the moving window before calculation currently if you have very small deviations (close to 0), it is difficult to distinguish between anomalous data, so it would be better to have a larger window size for those kinds of data expected output fields timestamp, current value, moving average, moving standard deviation, upper limit, lower limit, total anomalies, and anomaly field replaces current value, if detected anomaly detection parameters parameters details control chart mode? this parameter enables or disables the control chart mode, which bypasses the calculation for adjusting the standard deviation window and allows for simple upper/lower limit checks deviations controls how strict the “normal” range is standard deviation is a simple measure of spread low means values are close together and high means they’re more spread out this uses bessel’s correction ( n−1 ), so the input must contain at least two values example \[1, 2, 3, 4] → mean 2 5 , std ≈ 1 291 we treat values as normal if they’re within n standard deviations of the average values outside are flagged common choices 2 (stricter) or 3 (more forgiving) window size this represents the window in which the calculations are performed, measured in seconds it determines the number of data points included in the rolling window for observing average and standard deviation pass through value this parameter allows you to specify how to handle anomalous data, such as replacing the current value with an anomaly field if detected note when creating an analytics flow with anomaly detection processor, refer the docid\ ci 9aglo2iwr2ot7v0tvf guide for more details