For the past 15 years the cybersecurity industry has been stuck in a paradigm.  That might sound provocative given how many new technologies have been introduced to deal with the massive increase in cybercrime during the same period.  But the fact is that even though we have evolved from SIEM, to Log Management, to Big Data Analytics, and most recently, to Behavior Analytics, we have still been operating under the same set of assumptions.
That paradigm is finding Needles in Haystacks. It fundamentally is always looking for “bad” events in the terabytes of data.  

Even though we have built so many security technologies in last 15+ years, the reality is that we still see breaches in news headlines every few months, the good guys are still losing against attackers in the cyber war with this approach. 146 days is the mean time to detect. 529 million records are breached per year. 76% of enterprises consider threat visibility the greatest gap in security.  
If the definition of insanity is continuing to do things the same way and expecting different results, shouldn’t we be changing up how we approach this problem? In order to win this war against attackers, we need to change our strategy and play to our strengths against the attackers, instead of fighting this war by only chasing known attacks.

We propose a new paradigm:  “Removing Haystacks to Find Needles”

An analyst could not possibly know all the new and unknown attacks out there as by the time we find out about the attacks (known) attackers move on to the newer (unknown) attacks. However, the analyst certainly knows about all the known good in his environment. This tribal knowledge can be used to separate out the new unknowns from the known.

At the same time, no external attacker knows more than an analyst about his own environment. The reality is that most of the data in an environment is good and only 1% of it are potential indicators of breach. We need to play to that analyst’s tribal knowledge as our strength to identify all known good (99%), and by doing so anything unknown (1%) would surface up to the top for her attention. 

Removing haystacks means automating the processing of known good events from the consideration set such that new unknowns or needles are highlighted automatically at the top.  

Finding Needles Has Failed Because Analysts Don’t Scale

If there is a single factor that has led to the failure of successive cybersecurity technologies to keep up with the increase in threats it is scale. Events number in the billions, and alerts from those events number in the thousands. All the technologies we’ve been deploying until now try to find and deal with infinitesimally small bads in a sea of goods. As a result, all of these systems overwhelm analysts with false positives.

  • SIEM systems used fixed rule sets to identify known bads, but throw off so many alerts that must be reviewed by analysts that they are constantly overwhelmed by false positives. Plus, these rules are statics and hard to maintain.
  • Big Data Analytics systems still require analysts to run far too many reports to adequately handle the volume of data that needs to be evaluated. Furthermore, they are still heavily dependent on the analyst to mentally correlate all the evidence to make a decision. 
  • Behavioral Analytics systems, the most recent wave of security technology, still identify all kinds of anomalies (not necessarily threats) via a pure AI approach that is missing critical tribal knowledge that the analyst has about his environment. As a result, they end up generating too many false positives, again placing the decision making burden on the analysts. 

Many of these systems attempt to replace human analysts but ultimately generate more analyst work because they are missing context and analyst’s intelligence. On the other hand, there is a scarcity of analysts in the industry and existing analysts cannot scale. Organizations are struggling to hire and retain these experts. In this situation, how do we expect to ever win this war against attackers?

Play to Analyst Strengths Instead

What all of the security systems up until now have in common is that they are not designed to capture the most valuable contextual information analysts have available.
What do human analysts know better than any system or, more importantly, any intruder? They know their own environment. They know the enterprise context. They have an intuition about how their system operates and what is normal versus what is new and questionable in their environment. In other words, they know their haystack better than any attacker or vendor out there.
So, what if we built technology that helps analysts identify and remove haystacks as quickly as possible and focus instead on the needles proactively, the highest possible value-add activity they can engage in? This new technology needs to put analysts at the center and assist them with machine learning and automation to speed up and scale their decision making process.

The Technology to Play to Analyst Strengths at Scale

Recent developments make it possible to scale the analyst’s intelligence far more effectively than has been the case until now.  In particular:
1)  It is possible to use analyst-driven machine learning to capture analyst knowledge about known good events and identify them reliably.
2)  Based on that identification, it is possible to automate the elimination of known good events from the consideration set and free analysts to focus on new unknowns which could have hidden attacks.

Using AI to Identify Threats More Rapidly

The security systems currently in use are still chasing the old paradigm of trying to find known attacks or highlighting all anomalies being observed, and they most certainly do not use "haystacks first" approach. 
But it is now possible to feed the analysts’ tribal knowledge about context and environment easily into the machine learning system with an intuitive “feedback loop”, where the analyst can quickly review the system’s results and provide feedback that the system learns and improves from. This allows a properly designed system to adapt and evolve flexibly as context and environment change.  It also means such a system gets very good at making the haystack much, much smaller.

Using Automation to Scale

Automation, enhanced with analysts’ contextual intelligence and machine learning, can process large volumes of data quickly to eliminate the known good, and in the process unveil the unknown threats hiding in your haystack. As a result, the analysts shift their focus to investigating the much smaller data set of unknowns, which is more likely to contain real threats. 

A key element of the automation has to be a system that is easy to update, tune, and maintain and is built on a large scalable platform to replicate analyst’s intelligence.  


It does not take a secops ninja with 15 years’ experience to recognize that the way we try to funnel incidents out of our data today is broken.  The results, such as they are, speak for themselves:  no secops pro believes they are finding all of the threats, and by the time they do find threats the attacker has been dwelling in their environment for months.

We need solutions that put analysts in the driving seat with machine learning and automation as assistants to speed up their investigation process and scale their expertise.  Let’s move to a different approach, one with a greater impact made possible by new technologies and perspectives.  Let’s remove haystacks to find needles!

If you would like to learn more about this new technology and approach, reach out to us. We would love to hear from you and discuss further.