While SIM and other security analytics products are able to detect and alert on “known” threats, they are ineffective at recognizing and alerting on threats that the system does not already know how to detect. (If you want to learn more about why that is, read Monica Jain’s blog, 5 Key limitations of doing Threat Detection with Rules)

The simple truth about threat detection is that the intelligence exists today to detect and prevent most threats: that intelligence resides in the form of cyberhunters (or cyberanalysts). The problem is, these skilled human resources don’t scale well—and we do need to scale, as there is about 100x-1000x more data at large enterprises than their cyber hunters and analysts have the bandwidth to investigate.

The solution is to capture that intelligence, skills and intuition/context, automate it using software, and put it to work combatting the growing volume and complexity of today’s threats. That concept is at the core of Security Intelligence Automation (SIA).

SIA relies on 5 key innovations to automate human security intelligence:

Key 1: Create a repository of security domain knowledge

Domain knowledge is the understanding of what event data means and how it relates to the real world. Security analysts bring the breadth and depth of their personal experience in the industry, function, and role to bear on the decision-making they have to do as part of alert triage and cyberhunting.

Automated threat detection won’t be able to match the effectiveness of a human cyber analyst unless it has access to a similar knowledge base and can utilize that as part of the decision making process.

With automation, it is entirely possible that the system can access a much deeper set of domain knowledge than any one person can hold in his head—much like Google or Wikipedia.

Key 2: Capture investigative expertise

Not only do security analysts have a semantic understanding of what the data means, they know how to tie that with other pieces of data to gather much better context on the basis of which they need to make their decisions.

By capturing and applying this expertise, it is possible to rank threats with an effectiveness comparable to a seasoned human analyst.

SIA can help us capture the investigative process with very little overhead beyond tracking the steps that a cyber analyst performs daily while investigating and triaging alerts to determine whether an incident is real or not.

Key 3: Reduce complex events into more generic events

A critical tool that many analysts apply to the problem of threat detection is the ability to group events into “similar” events—and then reason over those similar events. This is what a reduction is; and it’s called a reduction because it can map a very large number of similar events to a few more generic event categories.

In a reduction, when an event has a higher rank in the system, all the events that map to it get ranked more highly because they are similar. SIA products should provide a powerful library of reductions that analysts can use out of the box. It should also make it very easy for analysts to add their own reductions to the library.

Key 4: Incorporate human classification and prioritization via smart tools

At its core Threat Detection is a Threat Prioritization or Threat Classification problem. We have shown how rules barely scratch the surface of that problem and look for only the “known bad” events, instead of trying to catch both the “known” and the “new unknown” bad activity. That job is often left to cyber hunters and cyber analysts.

An SIA product should offer these cyber hunters and cyber analysts tools to explore a set of security events with the goal of classifying these events in a methodical way, one backed by evidence. When they do so, automation tools capture the process and it becomes part of the algorithm. The workflow can be applied to automatically prioritize future events, or other ones from the past.

Key 5: Processes and assimilates analyst feedback easily

Security Monitoring has to succeed in a constantly changing environment. The threat landscape is never static. Also, a security analyst’s understanding of the environment that he operates in keeps getting refined. All of this happens very naturally. However, adapting the threat detection logic in our current systems is not similarly seamless.

Taken together, these five concepts constitute Security Intelligence Automation, a technology that can dramatically improve your analysts’ productivity (by up to 300%) while equally dramatically driving down your time to threat discovery and resolution.