Machine Learning can appear in many different forms and guises, but a general definition of Machine Learning usually incorporates something about computers learning without explicit programming and being able to automatically adapt. And while Machine Learning has been around for decades as a concept, it’s become more of a reality as computational power continues to increase, and the proliferation of Big Data platforms making it easier to capture floods of data. These developments have made ML practical and garnered a lot of interest, as evidenced by the large number of articles in the last two years surrounding AI and machine Learning

However despite all this, the adoption of this Machine Learning is still relatively low amongst companies in the tech landscape (Gartner estimating that fewer than 15 percent of enterprises successfully get machine learning into production). And even when you hear about Company X adopting a machine learning strategy, it’s often conflated with another strategy or service within that company, and not truly realizing the automated ‘adaptiveness’ inherent within ML.

Those companies that do realize a proper machine learning strategy, understanding and grooming their data as well as identifying the appropriate model/s can see real benefits to their operations, which is why DOSarrest has been developing such a strategy over the last year.

Here at DOSarrest, we’ve been focusing on building an Anomaly Detection engine, focusing on the constantly evolving sophisticated application layer attacks. We collect huge amounts of data from disparate sources (e.g. Customized web logs, snmp and flow data, IDS logs, etc.), even when customers are not under attack. This provides an opportunity to identify baselines even in a multi tenant environment. As you would expect, there is a high degree of cardinality within some of the data fields, which can be challenging to work with when working with data in motion, but can have great benefits. With these huge structured data sets, we are able to identify KPI’s (Key Performance Indicators) and statistics that can be leveraged by the engine to identify anomalous behavior and brought to the attention of the Security Ops team, who are then able to investigate and act on the identified pattern. The engine continues to refine the probability of a metric, becoming more accurate over time in determining the severity of an anomaly.

The strategy holds great promise, and further developments and refinements to this model will continue to evolve the best Security Operations Center in the business.

A more detailed view of an anomaly - this shows a single IP requesting more than 60 times more frequently than a normal visitor.

This screen gives an overview of any anomalies, organized by relevant factors. In this case the remote IP address of the requestor.

Jag Bains

CTO, DOSarrest Internet Security

Source: https://www.dosarrest.com/ddos-blog/machine-learning-in-the-dosarrest-operations