For the past 15 years the cybersecurity industry has been stuck in a paradigm. That might sound provocative given how many new technologies have been introduced to deal with the massive increase in cybercrime during the same period. But the fact is that even though we have evolved from SIEM, to Log Management, to Big Data Analytics, and most recently, to Behavior Analytics, we have still been operating under the same set of assumptions.

That paradigm is finding Needles in Haystacks. It fundamentally is always looking for “bad” events in the terabytes of data.

Even though we have built so many security technologies in last 15+ years, the reality is that we still see breaches in news headlines every few months, the good guys are still losing against attackers in the cyber war with this approach. 146 days is the mean time to detect. 529 million records are breached per year. 76% of enterprises consider threat visibility the greatest gap in security.

If the definition of insanity is continuing to do things the same way and expecting different results, shouldn’t we be changing up how we approach this problem? In order to win this war against attackers, we need to change our strategy and play to our strengths against the attackers, instead of fighting this war by only chasing known attacks.

We propose a new paradigm: “Removing Haystacks to Find Needles”

An analyst could not possibly know all the new and unknown attacks out there as by the time we find out about the attacks (known) attackers move on to the newer (unknown) attacks. However, the analyst certainly knows about all the known good in his environment. This tribal knowledge can be used to separate out the new unknowns from the known.

At the same time, no external attacker knows more than an analyst about his own environment. The reality is that most of the data in an environment is good and only 1% of it are potential indicators of breach. We need to play to that analyst’s tribal knowledge as our strength to identify all known good (99%), and by doing so anything unknown (1%) would surface up to the top for her attention.

Removing haystacks means automating the processing of known good events from the consideration set such that new unknowns or needles are highlighted automatically at the top.

Finding Needles Has Failed Because Analysts Don’t Scale

If there is a single factor that has led to the failure of successive cybersecurity technologies to keep up with the increase in threats it is scale. Events number in the billions, and alerts from those events number in the thousands. All the technologies we’ve been deploying until now try to find and deal with infinitesimally small bads in a sea of goods. As a result, all of these systems overwhelm analysts with false positives.

  • SIEM systems used fixed rule sets to identify known bads, but throw off so many alerts that must be reviewed by analysts that they are constantly overwhelmed by false positives. Plus, these rules are statics and hard to maintain.
  • Big Data Analytics systems still require analysts to run far too many reports to adequately handle the volume of data that needs to be evaluated. Furthermore, they are still heavily dependent on the analyst to mentally correlate all the evidence to make a decision.
  • Behavioral Analytics systems, the most recent wave of security technology, still identify all kinds of anomalies (not necessarily threats) via a pure AI approach that is missing critical tribal knowledge that the analyst has about his environment. As a result, they end up generating too many false positives, again placing the decision making burden on the analysts.

Many of these systems attempt to replace human analysts but ultimately generate more analyst work because they are missing context and analyst’s intelligence. On the other hand, there is a scarcity of analysts in the industry and existing analysts cannot scale. Organizations are struggling to hire and retain these experts. In this situation, how do we expect to ever win this war against attackers?

Play to Analyst Strengths Instead

What all of the security systems up until now have in common is that they are not designed to capture the most valuable contextual information analysts have available.

What do human analysts know better than any system or, more importantly, any intruder? They know their own environment. They know the enterprise context. They have an intuition about how their system operates and what is normal versus what is new and questionable in their environment. In other words, they know their haystack better than any attacker or vendor out there.

So, what if we built technology that helps analysts identify and remove haystacks as quickly as possible and focus instead on the needles proactively, the highest possible value-add activity they can engage in? This new technology needs to put analysts at the center and assist them with machine learning and automation to speed up and scale their decision making process.



The Technology to Play to Analyst Strengths at Scale

Recent developments make it possible to scale the analyst’s intelligence far more effectively than has been the case until now. In particular:

1) It is possible to use analyst-driven machine learning to capture analyst knowledge about known good events and identify them reliably.
2) Based on that identification, it is possible to automate the elimination of known good events from the consideration set and free analysts to focus on new unknowns which could have hidden attacks.

Using AI to Identify Threats More Rapidly

The security systems currently in use are still chasing the old paradigm of trying to find known attacks or highlighting all anomalies being observed, and they most certainly do not use "haystacks first" approach.

But it is now possible to feed the analysts’ tribal knowledge about context and environment easily into the machine learning system with an intuitive “feedback loop”, where the analyst can quickly review the system’s results and provide feedback that the system learns and improves from. This allows a properly designed system to adapt and evolve flexibly as context and environment change. It also means such a system gets very good at making the haystack much, much smaller.

Using Automation to Scale

Automation, enhanced with analysts’ contextual intelligence and machine learning, can process large volumes of data quickly to eliminate the known good, and in the process unveil the unknown threats hiding in your haystack. As a result, the analysts shift their focus to investigating the much smaller data set of unknowns, which is more likely to contain real threats.

A key element of the automation has to be a system that is easy to update, tune, and maintain and is built on a large scalable platform to replicate analyst’s intelligence.


It does not take a secops ninja with 15 years’ experience to recognize that the way we try to funnel incidents out of our data today is broken. The results, such as they are, speak for themselves: no secops pro believes they are finding all of the threats, and by the time they do find threats the attacker has been dwelling in their environment for months.

We need solutions that put analysts in the driving seat with machine learning and automation as assistants to speed up their investigation process and scale their expertise. Let’s move to a different approach, one with a greater impact made possible by new technologies and perspectives. Let’s remove haystacks to find needles!

If you would like to learn more about this new technology and approach, reach out to us. We would love to hear from you and discuss further.


Related Posts

May 20, 2022 Willy Leichter

Automating Threat Detection: Three Case Studies

Demystifying the technology with case studies of AI security in action Many automation tools, such...

Learn More

May 17, 2022 Willy Leichter

It's Time to Put AI to Work in Security

While we’ve been talking about and imagining artificial intelligence for years, it only has...

Learn More

May 15, 2022 Tessa Mishoe

LogicHub Security RoundUp: May 2022

Hello, and welcome to the latest edition of the LogicHub Monthly Update! Each month we’ll be...

Learn More

May 9, 2022 Tessa Mishoe

Bad Luck: BlackCat Ransomware Bulletin

Blackcat Ransomware On April 19th of 2022, the FBI Cyber Division released a flash bulletin...

Learn More

May 6, 2022 Kumar Saurabh

Let Humans Be Humans and AI Be AI

LogicHub’s unique decision automation technology can build clients the ultimate security playbook...

Learn More

May 3, 2022 Kumar Saurabh

How to Build a Threat Detection Playbook In 15 Minutes or Less

Automating a threat-hunting playbook with the help of AI Many threat-hunting playbooks we build for...

Learn More

April 29, 2022 Tessa Mishoe

Integrating Better: What Can Integrations Do For Me?

Introduction Within the realm of security, there are many different toolsets and opinions on what...

Learn More

April 27, 2022 Willy Leichter

Beyond No-Code: Using AI for Guided Security Automation

SOAR Playbooks Outside of football, the term “playbook” is well understood by a relatively small...

Learn More

April 21, 2022 Willy Leichter

Goodbye Lonely SIEM, Hello MDR

When updating your systems from a pure Security Information Event Management (SIEM), choosing the...

Learn More

April 15, 2022 Tessa Mishoe

LogicHub Security Roundup: April 2022

Hello, and welcome to the latest edition of the LogicHub Monthly Update! Each month we’ll be...

Learn More