By Ira Cohen
Most companies are sitting on top of a veritable gold mine. Gigabytes of digitized time-series data are just ready and waiting for someone to mine that million-dollar nugget that aids in the creation of the next successful service, product, or solution. Yet, ironically, for most companies the challenge isn’t in getting access to the data, it is analyzing this vast quantity of information in such a way that they can gain unique and actionable insights. With large-scale, complex data, achieving real-time insights can be a struggle, particularly with conventional methods, such as human analysis, dashboards, and alerts.
In this article, we’d like to share with you how the autonomous anomaly detection techniques developed by Anodot extract powerful business insights from your data.
Leveraging Time-series Data
Time-series data is simply a sequence of values captured and indexed in time order. Each data point typically contains two items: the moment at which the data was measured and the actual value of the data, which when plotted can give an idea of past trends and behaviors. With the right algorithms, that time-series record can be used to predict with reasonable certainty what future trends might hold. In anomaly detection, time-series data is used to leverage those educated predictions to target outliers in the data. These outliers—or anomalies—can represent that gleaming nugget of information that pinpoint an emerging trend worth millions to a company—or at the same time, find the smallest glitch that has the ability to cause a company significant financial or reputation damage.
All industries possess time-series data, typically in the form of key performance indicators (KPIs) that can be captured and analyzed to extract real-time and predictive insight. KPIs can include webpage usage, mobile device data, sales figures, customer demographic data, geographies, infrastructure and equipment production information, social media, and weather, just to name a few.
Anodot’s real-time anomaly detection system uses time-series data by first classifying it and then selecting the right algorithm to analyze the data’s behavior. The Anodot system then predicts how the data points should be temporally distributed, given the model. A statistical test is then applied to test future data points based on the expected distribution. If a value falls outside that distribution, it is flagged as a potential anomaly.
Some forms of time-series data have cyclic patterns—referred to as “seasonality,” with periods spanning hours, days, months, or even years. And sometimes, multiple seasonal patterns are present in a time series. There are several methods to analyze seasonal time-series data. The Fourier Transform and Serial correlation (also known as auto-correlation of signals, auto-correlogram, or ACF) are two methods, but both present problems. Anodot uses its own proprietary algorithm, named “Vivaldi” for seasonal time-series analysis. Vivaldi uses smart subsampling, reducing the required computation. By applying this method on multiple filtered versions of the same time series, multiple seasonal time-series patterns can be detected.
The Benefit of a Hybrid Multivariate and Univariate Anomaly Detection
With millions of time-series metrics to analyze, it becomes easy for an anomaly detection system to fall prey to false positives and an overload of alerts. Anodot believes that both the big picture story and the granular per-metric anomaly detection must be preserved. We refer to this as “conciseness” and it is a requirement of any large-scale anomaly detection system.
With Anodot’s system, conciseness is achieved through univariate and multivariate anomaly detection techniques. A univariate anomaly detection system looks for anomalies in each individual metric; multivariate anomaly detection uses a single model for all the metrics in the system. Using this method, first the Anodot system detects anomalies from individual metrics, and then groups the anomalies together based on those metrics that are related, condensing what could be a flood of alerts into a smaller manageable series of incidents. Conciseness is achieved using this hybrid approach, ensuring scalability and simplicity.
The Importance of Truly Automating Automated Anomaly Detection
Throughout our recent posts, we have discussed the topic of scalability. From an automated anomaly detection technique perspective, scalability is essential. If you represent a very small company that is managing just a couple of business metrics or KPIs, and you don’t expect your company to grow or change much, then you probably don’t need to worry about scalability. However, if your company is like the vast majority of organizations out there, then you’re probably dealing with hundreds, thousands, or even millions of data sets, and data volumes are growing every day. Having an anomaly detection system that scales to your needs is critical. And automation is an indispensable component of scalability.
To understand why scalability is so important, let’s look at three key components of real-time autonomous anomaly detection: (1) “Detection” identifies anomalies and pinpoints their source; (2) “Ranking” grades the anomalies based on their significance; and (3) “Grouping” places the anomalies in a category for concise reporting. For a small- to medium-sized business (SMB) the detecting, ranking, and grouping process using only manual human analysis can be cumbersome. For example, to manually process 1000 metrics, an SMB would need 10 people examining 10 metrics each constantly to ensure real-time awareness. As the number of metrics increases, so does the staff requirement; 10,000 metrics would require 100 people and 1 million metrics 10,000 people.
In terms of ranking and grouping, the human factor also creates inherent problems with quantitative consistency; inconsistency can result in irrelevant anomalies floating to the top and important anomalies being missed.
Even if a company could afford a huge team of analysts, the communication overhead between the team members as they discover, rank, and group anomalies for concise reporting increases much quicker than the number of analysts, according to Metcalfe’s Law. For example, if 1,000 people are monitoring a total of 100,000 metrics, this group would need 499,500 communication connections to ensure they could effectively discuss their ranking and grouping decisions. Real-time manual anomaly detection is simply humanly and financially unfeasible.
Most companies can’t actually afford to hire thousands of analysts to sift through data before every important business decision. Automation is the key to scaling. Whether you have one thousand metrics or one million, Anodot’s real-time, large-scale automated anomaly detection system offers automation at every step in the process. By discovering outliers in vast amounts of time series data, the machine learning system turns information into valuable insights that can be leveraged to develop business strategies, identify issues, and correlate trends, eliminating business insight latency and supporting rapid business decisions.
Anodot was founded in 2014, and since its launch in January 2016 has been providing valuable business insights through anomaly detection to its customers in fintech, ad-tech, web apps, mobile apps, e-commerce and other data-heavy industries. Over 40% of the company’s customers are publicly traded companies, including Microsoft, Waze (a Google company), AppNexus, and many others. Anodot’s real time business incident detection uses patented machine learning algorithms to isolate and correlate issues across multiple parameters in real time, supporting rapid business decisions. Learn more at: http://www.anodot.com/.