And it is not always about moving off the large vendors. Many teams are rethinking how they implement SIEM in the first place.

Because we now have so many tools that break down SIEM functionality, the evaluation is around what makes sense as a modern architecture. What stays in one platform, and what gets decoupled.

And yes, I think AI triggered this. Or I would say the evaluations for AI SOC started it.

Many teams started evaluating AI SOC to see how they can automate their SecOps faster and simpler. Then many came to the same realisation: you can’t just add AI and expect it to work. You need to change the way you work.
And once you change the way you work, you end up re-evaluating your whole security stack. SIEM included.

We see it on the vendor side too, as they build what practitioners ask for. SIEM vendors are rushing to add decoupled options, be more flexible, and ship AI SOC capabilities and agent builders. AI SOC vendors are moving toward becoming SIEM.

The two sides are meeting in the middle, and the SIEM label alone tells you less and less about what you are buying.

Migrate the SIEM before you change how you work, and you end up with the same SOC on a new platform.

Have a siemless weekend ahead!
Read the original post and the comments on Li

You’re probably thinking, “Yes Filip, anything new here?” Not much, honestly. But I wanted to clear something up.

The whole buzz and fuss around context DBs and context graphs is not new. The idea is not new. What changed is who consumes it, and that AI can now help build it from messy sources. Before, this context was built mainly for humans, and in the best case for some SOAR automation.

Product Updates

Agentic case management is the next layer of SecOps to get rebuilt. Think ITSM, built natively for security. Treat links every tool, and your policy, through one ontology of identity, network, and assets, so everything about the same asset lands in one case. General-purpose LLMs handle reasoning. Purpose-built Cyber Language Models handle the security context and the decision. When an employee asks to approve a new AI tool, Treat checks their identity, access, risk, your policy, and past decisions, then recommends the path: auto-resolve, your approval, or page on-call.

Treat measures success as Mean Time to Trusted Action.

Don’t Have to Choose Between Human Oversight and Automation.

Humans at the Helm means security teams can adopt and expand automation progressively, as verified outcomes accumulate, establishing credibility.

SOC teams oversee automations, configure agents and playbooks, and control investigations, evidence validation, and response. They make the critical decisions and can automate triage, workflows, and playbooks partially or fully, in stages or all at once, based on the evidence they need to build trust.

This is governed automation. This is the AI SOC done right.

No CMDB and IdP, no magic

So if you’re evaluating tools in this space, first things first. As mentioned, if you have a bad CMDB (and I’m not sure how many people can say they have a good one) and no proper inventory of your infrastructure and identities, then no AI SOC, ISOC, XDR, or AI-this-and-that will automagically save you.

To be fair, AI SOC tools can get part of the way there. They can query your EDR to learn about the host and your IdP to learn about the user. But in most cases that only gets you the technical facts. It doesn’t tell you who owns the box, how critical it is, or whether that service account is supposed to be doing what it’s doing. That’s business context, and it usually lives in a CMDB nobody updates, a spreadsheet, or someone’s head.

Institutional and tribal knowledge

OK, maybe there’s a tool that gathers all this institutional knowledge for you. And yes, there are some on the market. But even there, it’s a fine line. How much do you want to monitor your employees? To be honest, in many cases capturing all that knowledge requires some level of monitoring of how people work.

And if you’re in Europe, this isn’t only an ethics question. It’s GDPR. In some countries, like Germany, works councils also have a say on monitoring tools. So check with legal before you sign anything that watches your analysts work.

Then there’s tribal knowledge. Sorry to tell you, there is no easy way to capture that one. Unless you want to try extortion :D

The only realistic way is slowly, through feedback. Every time an analyst corrects the AI, overrides a verdict, or adds context it missed, that’s tribal knowledge leaking into the system. If your AI SOC has no proper feedback loop, it will never learn the stuff that isn’t written down.

Humans are bad at documentation

Also keep in mind how most of this was built in the past. It was done by humans or by automation. And text generation wasn’t great back then, unless you invested heavily in NLP/ML stuff.

Humans are bad at documenting stuff. We’re even worse at keeping that documentation accurate. At one of my previous jobs, we had a person whose only job was writing and maintaining documentation. It was pretty good while they did it. Then they left and it all fell to pieces again.

And a lot of what we call process isn’t documented at all.

Your alert history is not ground truth

Same with alerts. If you’ve ever done QA on past alerts closed by humans, you know what you get. The usual one-liners: “Normal activity.” “Expected.” “Closing as FP.” Incidents might be better, because someone was forced to do a post-mortem and write an actual executive summary of the investigation.

Here’s why this matters for AI SOC. If the tool uses your past dispositions as memory or as ground truth, it inherits those labels. “Closing FP” with zero reasoning becomes something the AI learns from. Garbage labels in, confident garbage out.

So before you let any AI learn from your history, ask yourself honestly how much of that history you’d trust.

Ground rules

So let’s set some ground rules. Or just state the obvious, in case you missed the memo.

1. No good asset and identity context, no AI SOC magic. The AI can pull a lot from your EDR and IdP, but it won’t know who owns the box, how critical it is, or whether that service account is supposed to do that. There’s no magic button for this. You need to organize and document your data well.

2. Your documentation was built for humans, and often badly, by humans. Once you add AI, it will take time for the AI to document things properly and turn them into a good, machine-readable format. (Check my older blog on why your playbooks were not created for machines: [link])

3. LLMs don’t “know” threats they weren’t trained on. ML can flag abnormal behaviour, but only against a baseline. If you don’t know what normal looks like in your org, and you don’t have good change and exception management, you’re doomed. You’ll get flooded with FPs, or even worse, miss the TPs.

Final Thoughts

AI SOC doesn’t replace the basics. It runs on them.

Fix your asset and identity data first. The AI part gets a lot easier after that.

SecOps Unpacked Webcast Episode on Machine Speed vs Machine Focus


Check out our SecOps Market Landscape tracker and evaluation frameworks

SecOps Market Landscape

Yes, we have new ways of running SIEM. But each implementation comes with tradeoffs, and it doesn’t automatically mean you will get better results. It highly depends on the environment where you run it.

Hot, warm, and cold storage? We’ve had those for years. Separating collection, storage, search, detection, and response? Also familiar concepts. And we don’t need MCP to run SIEM queries.

What is changing is where vendors put these capabilities, how they package them, and who takes responsibility for making everything work together.

You can bring more capabilities into one platform. You can decouple pipelines, storage, detection, and response. You can combine both approaches.

I think the debate in many cases is: do you want real time SIEM, or do you settle for near real time? Measure that against your actual use cases. Neither an integrated platform nor a distributed architecture guarantees the answer.

An integrated platform can simplify operations while increasing dependence on one vendor. A decoupled stack can give you more control while leaving your team responsible for integrations, schemas, permissions, and troubleshooting. Decoupling can reduce vendor lock-in, but it doesn’t magically remove it.

AI adds more queries, more tool access, and potentially more response actions. That matters for architecture and operations. It doesn’t make the underlying storage and detection concepts new.

Today, you can buy a data pipeline with SIEM capabilities. A data lake with SIEM capabilities. A SIEM bundled with EDR, NDR, SOAR, and AI SOC capabilities. Even an AI SOC platform with SIEM capabilities.

SIEM has always been an architecture problem. The packaging keeps changing.

Also we are having interesting discussion next week Machine speed vs. Machine focus

Register Here

Have a siemless weekend ahead!

Read the original post and the comments

But what do you think of Microsoft going with the ISOC term? Was it just one of those “why come up with a unique name when we can take Gartner’s new term” moments? 😅

Are you ready for the ISOC Copilot?

Have a siemless weekend!

Read the Linkedin post and the comments

Is your SecOps toolset changing because the roles are changing? Or is it the other way around?

Most vendors promise the same thing today: triage is fully automated, so your team can focus on what matters. But your team was already focusing on what matters. The low severity alerts were bulk closed anyway. The high and critical alerts got looked at. The low severity alerts? They were bulk closed, suppressed, or left in the queue until someone wrote a rule to close them.

So what changed? You have a bit more of the boring tasks automated. And maybe better coverage on the low and medium severity alerts you used to ignore. That’s a real improvement I wont argue that we all know attackers hide in the noise (not every time, but when we catch ones that does we turn that into a marketing story)

My take: the big shift is in the roles. Not because you don’t do triage anymore :) But because building got a lot easier.

Product Updates

Latest from Fig Security

Fig sits above your existing stack and maps how data flows from source to detection. It works with any data source, SIEM, pipeline, or data lake. One read-only API, live in days.

Catch, Trace, and Fix Breaks
Fig checks every change before it ships and keeps checking in production. If a log source, schema, or parser change breaks a detection, Fig alerts you in Slack, email, or Jira, with the root cause and a fix.

Build, Test, and Ship Changes
Tell Fig what you want to change or know. It returns a detection, parser, query or configuration that’s ready for production and tuned, tested, and validated against your environment. All in minutes, not weeks.

Don’t break for a change.

New release : Sekoia Elevate

Sekoia Elevate is the AI agent layer of the Sekoia agentic SOC platform. It investigates every alert and case end to end, starting the moment it’s created.

Investigation and Verdict
The agent pulls raw logs, detections, runbooks and Sekoia’s in-house threat intel, then pivots through the context and tests hypotheses. Analysts get an audit-ready verdict with confidence, findings, reasoning and the queries behind each finding, and can override it.

Runbooks and Sovereignty
Generates investigation runbooks for every detection rule, including custom ones. AI runs on Sekoia-hosted infrastructure, with no data sent to external LLM providers. Built for enterprise SOCs and multi-tenant MSSPs.

Every alert investigated. Every decision explained.

People build machines, machines do the job

Anton Chuvakin sums up SecOps maturity in his blog:

Today, humans build the machines with the help of other machines, and then the machines do the heavy lifting.

The building part got easier. You don’t need a months and a dedicated SOAR engineer to ship couple of workflows. So you will build more stuff that runs automatically, and some of it autonomously. And that means many roles that were pure SOC analyst will move to security engineering.

Yes, if you go with a vendor, some of this comes pre-built. You don’t build or manage the triage agents yourself. But someone still has to fine-tune them, feed them context, and check if they’re right. That’s engineering work too.

So what does that work look like? Let’s break it down with the SecOps AI Shift Map: left is data and detections, middle is investigation, right is response.

Far left: the data

Your job here is to make sure you connected the right sources. In other words, you have the right data and you monitor it. Connecting tools, building data pipelines, writing parsers, normalizing fields. There’s a ton of good tooling out there for this.

Why does this matter more now? Because an agent can only investigate what it can reach. If your identity logs aren’t there, the agent doesn’t say “I’m missing identity logs.” It gives you a verdict anyway. Garbage in, machine speed AI-powered garbage out.

The basics still win here. Tag every event with an asset and a user. Know which sources support detection and which are only good for investigation after the fact. And expect parsers to break, because some vendor will change their log format on a Tuesday without telling anyone.

Left: detections and context

Detection engineering is a whole category on its own. Writing rules, testing them, tuning them, retiring the ones that only produce noise. No news there. What’s new is that AI triage now gives you feedback on every detection, including the low severity ones nobody looked at. So you finally know (easier and faster) which rules are noisy. The question is whether someone has time to fix them.

Next to it is context. The enrichment sources for your SIEM, your TIP, your CMDB, your IAM. Who owns this machine? Is this a service account or a human? Is this asset critical? Is there an SOP attached to this detection? How you manage and structure all of this is becoming its own job.

I think this becomes a responsibility on its own, and in some cases a full role, most likely in large enterprises and MDRs. Let’s call it Security Context Engineering. (Yes, another term. It’s a cybersecurity tradition at this point.)
Also some great read on the context piece discussed by Alankrit Chona

Why a role? Because agents reason on context. Good context, good verdict. Bad or missing context, and you get a confident wrong answer. Someone has to own that context the same way someone owns the detections.

I’d also put building agents and automations here. Either you build them because your vendor doesn’t ship pre-packaged stuff, or you maintain the ones you already have. Deterministic playbooks for the steps that need to run the same way every time. Agents for the ambiguous stuff that needs reasoning. And please, version control them.

And in the end you need something that monitors your SecOps pipeline itself. A log source goes quiet. A parser starts dropping fields. A detection stops firing. An agent’s accuracy slowly drifts. None of these throw an error. They just quietly make everything downstream worse. Someone has to watch the plumbing.

This whole left side of the house becomes more and more important. Every problem you fix here, the agents in the middle don’t have to work around.

Middle: investigation

This is where most AI SOC vendors started, because it’s the easiest part to show value. And this is where the analyst job changes the most.

The engineering work here is tuning triage and investigation agents, and QA. A lot of QA. Pulling closed alerts and checking if the verdict was right. Reading the agent’s reasoning and asking “would I have closed this?” Tracking how often analysts reopen something the agent auto-closed. Maybe this is the new analyst work.

And the new role here is the Security Eval Engineer. This person builds test sets from real past alerts and incidents, runs them every time the model, the prompt, or the vendor version changes, and tracks if accuracy goes up or down. Pair it with breach and attack simulation and you can check if the whole chain works, from detection to verdict.

As I said many times, machine speed means nothing if you close alerts wrong, faster. Accuracy and reliability are your new top KPIs. Not MTTR. Not alerts closed per hour. That’s how you gain trust, and without trust nobody will let the agents move to the right side.

Right: response

Here you have the remediation and mitigation agents. This is the hardest part to automate, and not because of the tech. It’s processes, change management, and the fact that nobody wants to give an agent a service account with full write access. Fair enough.

So the engineering work is:

• Aligning with your processes and procedures. The agent follows your IR process, not its own idea of what good response looks like.

• Threshold management. Which actions can run automatically, and which need a human to approve. Disabling a user on a test laptop is not the same as isolating a production database server. Write these rules down as policy, not tribal knowledge.

• Building reversible actions. Isolate a host, you can un-isolate it. Reset a password, the user gets a new one. Delete something, it’s gone. Start with the actions you can undo.

• Making sure your feedback loop works. Every incident should send something back to the left side. A detection to tune, a log source to add, context that was missing. Most teams write the lessons learned report and never look at it again.

What this means for your team

If you run a SOC today, a few practical things:

• Map who owns each part of the Shift Map. If nobody owns context or eval, that’s your gap.

• Move analysts toward QA, eval, and context work. They already know what a good investigation looks like. That’s exactly the skill you need to check an agent.

• Stop hiring for clicking through alerts. Hire for people who can build, test, and question the machines.

Final Thoughts

So yeah, this is how security engineering will work. Triage didn’t go away. The machines do it now, and your job is building, tuning, and checking those machines.

For the tooling side, we have the full mapping based on the Shift Map, organized by the outcomes you get from each tool

Machines do the job. People make sure it’s the right job.

Check out our SecOps Market Landscape tracker and evaluation frameworks

SecOps Market Landscape

Have a siemless weekend ahead!
Read the original post and the comments

Many SIEMs now have some form of context, entity or security graph.

User → logged into → Host → connected to → IP

But there is another layer that matters as we move toward autonomous SecOps:

Why is this normal in YOUR environment?

Security is full of weirdness. If everything was perfectly locked down, nobody could get any work done. 😄 So companies make exceptions. Then exceptions on top of exceptions.

> A developer has root access to one production server.

> Two servers invoke PowerShell in a way that would normally make your detections scream.

Your SIEM can observe these , your context graph can connect them. But does it know why they exist?

As you know usually that knowledge sits in a ticket, a Slack thread, docs/confluence pages, or the head of the analyst who investigated the same thing three times and knows “yes, this looks insane, but it is expected.”

The knowledge layer.

Security knowledge about your tools and controls, combined with business knowledge/intelligence about how your organization actually gets work done.

If we want an agent to call it benign autonomously, more logs aren’t enough. It has to do the legwork the analyst does today: reach out to people or other agents, retrieve the why, and store it.

But this is tricky. It is not something you gather once and trust forever. Things change, so the agent has to verify nothing changed before relying on it. And if verifying means asking the same people again, they get pinged for the 10th time about the same thing.

The fix is a good way of tracking exceptions, change management, who approved it and why, recorded where the agent can check it.

Some AI SOC vendors are already building organizational knowledge/memory into their platforms, and some SIEM vendors are starting to. It is early, but I wouldn’t be surprised if SIEM vendors race to own it over the next 2-3 years, organically or through acquisitions.

Be careful where you build it. Context/Knowledge lock-in might become more painful than data lock-in. Logs are easy to move. Years of knowledge about

If this becomes one of your most valuable datasets, I would want it vendor-agnostic and portable.

Have a siemless weekend ahead!

Originally posted on LinkedIn on 25 September 2026
Read the original post and the comments

Check out our SecOps Market Landscape tracker and evaluation frameworks

SecOps Market Landscape

A few weeks ago I wrote How Not to Kill Your AI SOC. The short version: an AI SOC is at least three technologies. Deterministic code, ML, and language models. If you route work to the wrong tier, it breaks in ways no accuracy number will show you.

This week two things landed that push the same argument.

Alex Beaver published The Asymptote of the AI SOC is Determinism. It’s the best write-up I’ve read on why better AI makes deterministic automation more important, not less. Read it.

And TypeSafe released Jev, a new frontier model whose whole design is “put a fast decision inside deterministic code.”

So yes, we needed a new frontier model to confirm we should write code first. I’ll take it.

Product Updates

Armor Detect by Arambh Labs

Armor Detect is a detection engineering agent that runs the full lifecycle, from coverage gaps to tuning.

Coverage and Creation
Finds your MITRE ATT&CK coverage gaps. Creates production-ready detections from plain English and adapts the rules to your actual telemetry.

Validation and Tuning
Backtests every detection against historical data before deployment, then ships it through CI/CD. Once live, it keeps tuning noisy or ineffective rules.

Detection engineering as an autonomous loop, not a never-ending backlog.

Intezer Workflows and Custom Agents

Intezer already investigates 100% of alerts and escalates fewer than 2% for human review. Two new releases cover after the verdict.

Intezer Workflows
Response automation native to the AI SOC, instead of a separate SOAR, tested before they run live. Triggers on an analyst’s click or autonomously on verdicts. Describe it in plain language and Intezer’s MCP builds it.

Custom Agents
Build your own agents on top of the ones Intezer ships. Each one runs your custom instructions, a trigger event or schedule, and the tools it can use. Teams run them for incident reports, tuning recommendations, threat hunting, case routing, and shift handoffs. Every run shows reasoning and tool calls, and Dry Run mocks write actions before go-live.

The asymptote

Alex’s argument comes from automation economics. Automation means paying more up front so each run costs less. That math works best for work that is frequent, stable and rule-based. Sounds like most of a SOC.

What stopped us before was the up-front cost. Engineers are expensive, and every exception needed a human to write it. So a lot of automation never got built, and agents looked like a good middle ground. Cheaper per case than an analyst, and they scale with alert volume.

But AI coding tools are killing that up-front cost. A good engineer with a coding agent builds playbooks much faster than before. So work that wasn’t worth automating deterministically suddenly is. And for repeatable work, a GPU running inference won’t beat a CPU running code on cost.

His conclusion: as more of an agent’s work becomes repeatable, it moves into deterministic code. Not to 100%, because agents still handle what code can’t. But that’s the direction.

He has a line from real POCs that every practitioner will recognize. He spent more time prompt-engineering the agent than it would have taken to build the logic deterministically. Been there.

This is the same loop I described in the last blog. If the LLM handles the same edge case again and again, codify it. Alex’s version just explains why that loop wins over time.

The step you couldn’t write in code

There’s a gap in that picture, though. Some SOC steps are repeatable but fuzzy.

Is this email trying to steal credentials? You answer this thousands of times a week, so it’s repeatable. But you can’t write a regex for it. So today it either goes to a rule or to an LLM call at frontier prices.

This is where Jev fits. Alex calls it the first model in a long time he can see being useful. Classification turns unstructured data into structured data, and that’s the most practical thing AI does for us.

Jev in 60 seconds

Jev is what TypeSafe calls a System One model. You give it a state (an alert, an email, a log line, a ticket) and a set of questions. It returns typed answers with probabilities. It doesn’t reason or generate text.

There are three question types, and each one maps to something your analysts do every day:

• Choice picks one option from a list. What type of alert is this?

• Score rates something against levels you define. How bad is this for our environment?

• Noul asks a yes/no question and returns the probability the answer is yes. Does this email ask for credentials?

TypeSafe says it’s 40x to 200x faster than frontier LLMs on these kinds of queries, with 70 to 500ms response times. Input costs $0.042 per million tokens, and output is free. Those are their benchmarks, not SOC data.

The important part: it’s a model that lives inside deterministic code. It doesn’t replace the code. So it fits the asymptote instead of fighting it.

Walking an alert through it

Take a user-reported phishing email.

Step 1: Deterministic. Parse the email. Extract URLs, sender, attachments and headers. Run reputation lookups, check SPF/DKIM/DMARC, and check your allow lists. The correct operation is known, so no AI needed.

Step 2: Jev. Ask a batch of yes/no questions in one call. Does it ask for credentials? Does it impersonate an internal team or a known vendor? Does the link text match the destination? Is it a newsletter the user subscribed to? The questions run in parallel, so asking 15 instead of 5 barely changes response time.

Step 3: Thresholds in your code. Jev returns probabilities, and your code decides what they mean:

answers = response.answers

creds = answers["asks_for_credentials"].noul
newsletter = answers["is_subscribed_newsletter"].noul

if newsletter > 0.9 and creds < 0.1:
    close_as_benign(alert)            # deterministic close
elif creds > 0.9:
    escalate_and_enrich(alert)        # deterministic escalation
else:
    send_to_llm_investigation(alert)  # the ugly middle

Step 4: LLM for the middle. Only the alerts Jev isn’t sure about go to reasoning. That’s where the LLM checks who else got the email, builds a timeline, and weighs evidence that points in two directions.

Step 5: Deterministic enforcement. Purges, blocks and account disables run through code with approval gates.

Deterministic code carries the volume. Jev handles the fuzzy but repeated calls. The LLM gets what’s left, which, if Alex is right, should shrink over time.

Other places I’d put it

Alert routing. A Choice across your use cases (phishing, identity, endpoint, cloud, DLP) sends each alert to the right playbook/agent.

Checking the LLM’s work. Does the agent’s summary support its verdict? Score it before anything acts on it.

Detection tuning. Log every probability. If a detection fires 3,000 times a month and almost all of it gets scored confidently benign, that detection goes on the tuning list.

Your pipeline has to be ready for it

Everyone is racing to ship agentic features. Not many are fixing the deterministic engine underneath.

That matters a lot for Jev. Its whole value is that it returns typed answers you can trust in code. If your SOAR passes loose JSON blobs between steps, you lose that value at the first hop. And a 100ms decision doesn’t help much if the playbook around it needs a container cold start to run.

So before you get excited about a fast model, check your plumbing. A good model on a bad foundation still performs badly.

Who is this actually for

This is a narrow tool, and I think that’s fine.

Most SOC teams won’t call Jev directly. Someone has to write the questions, set the thresholds, wire it into the pipeline, and test it on real data. That fits three groups:

• AI SOC vendors, where cost and latency per alert hit their margins

• Agent builder and SOAR platforms, which could offer it as a step type

• In-house teams with real security engineering capacity

For a three-person SOC, it’s something your vendor might use under the hood, not something you pick up on Monday.

What it doesn’t fix

Type-safe is not correct. Jev can’t return an invalid type. It can still return a wrong answer in a valid type, and a confident 0.95 on the wrong thing still closes the alert. Calibration is TypeSafe’s main claim, and that’s what I want to test on security data.

It doesn’t baseline anything. It only sees the state you give it. “Is this login unusual for this user?” still needs a baseline from the ML tier. Garbage state in, well-calibrated garbage out.

Don’t build an agent with it

Your timeline is full of Jev agents doing tasks at crazy speed. Alex puts it well: no, your “Jarvis” isn’t going to run entirely on Jev. A lot of the demos use it outside what it’s built for.

It has no reasoning and no text generation. It’s a decision inside code, not an analyst.

So, did you kill your AI SOC yet?

If you haven’t, here’s another chance not to.

I’m still testing Jev, so no results yet. I’ll be back with numbers.

Jev won’t investigate anything, won’t baseline your users, and won’t respond to incidents. It answers small questions fast and tells you how sure it is. In a SOC that’s a narrow job. But it’s the job that runs ten thousand times a day, and right now a lot of AI SOCs are paying frontier prices to do it.

The goal is still not to put AI everywhere in the SOC. It’s to know where not to put it.

Check out our SecOps Market Landscape tracker and evaluation frameworks

SecOps Market Landscape

And yes, don’t jump on me for the vendors in the visual. When I say SIEM, I include both the more traditional ones and the SIEM alternatives, next-gen, decoupled, integrated SecOps, whatever we decide to call them this week.

There is also a really cool blog from the Elastic team, a proper deep dive into how they run this internally, with some great insights. One that stood out was how much they improved agent accuracy, not by changing the model, but by giving the agent better context.

That part matters a lot.
Overall, don’t use agents for everything. Go deterministic where you can and where it makes sense. Agents are useful, but they are not some magical replacement for every workflow just because we discovered the word “agentic.”
They also touch on why dumping walls of AI-generated text in front of analysts is not particularly useful. But that is a topic for another Friday SIEM post, because the UX disaster we are creating around AI in SecOps deserves its own rant.

Overall, I think we will see most, if not all, SIEM vendors adding some form of Agent Studio.

What surprised me is that not many added this capability through acquisition. A lot of them seem to be building it in-house. I still suspect we will see at least one acquisition in this space, maybe even this year.

And just to separate two things that are getting mixed together a lot.

When I say AI SOC, I usually mean the pre-built stuff. Investigation agents, triage agents, response agents, all ready to go.

When I say Agent Studio, I mean the build-your-own part. You get templates, tools, connectors and some building blocks, but you can create or customize the agents yourself.

So yes, SIEM added SOAR, then AI SOC, and now Agent Studios as well.

Have a siemless weekend ahead!

Read the Linkedin Post and comments

I think more people are finally realizing how important detection engineering is. Bad detections create a ton of pain downstream. But if we go even further left, the data pipeline matters just as much. Garbage in, garbage out still applies, even if the garbage is now being investigated by an AI agent with expensive frontier model .

If I had to put the SecOps lifecycle on a scale of importance, I’d probably put more weight on the left side. The funny part is that the left side is also the quiet one. Alerts, investigations and incidents usually make it obvious when something is wrong.

Pipelines and detections, on the other hand, can be broken for weeks without anyone noticing.

In most SIEMs, you won’t find out until an alert is supposed to fire, assuming it ever does.
By default, not many SIEMs give you particularly good pipeline monitoring(SIEM vendors don't jump now to convince me yours has the best one), so you need continuous testing to catch broken ingestion, parsing or detection logic before the attacker does.

And this is where I think attack emulation, detection validation, pipeline monitoring and the broader SecOps resilience bucket become really important.

Because if you don’t continuously test whether your detections actually catch what they were designed to catch, you basically have two options: trust that they work, or let the attacker QA them for you.
Neither is a great strategy.

Same for pipelines. If logs stop arriving, fields change, parsers break or some connector quietly dies, you might still have a beautiful SIEM dashboard. It just happens to be beautifully wrong.

Attack emulation has obviously been around for a long time, so none of this is particularly revolutionary. What I think is changing is that the category starts to expand. Automated red teaming, BAS, attack emulation, detection validation and probably parts of pipeline validation start converging into something closer to a continuous SecOps resilience layer.

And I think that becomes even more important as the SOC gets more automated. If agents are going to investigate, triage and eventually make more decisions for us, someone still needs to make sure the foundations underneath them are not held together with duct tape and three broken API integrations.

Does attack emulation remain its own category, or does it eventually become a native part of the SIEM / SecOps architecture?

Have a great weekend ahead!

Originally posted on LinkedIn on 11 September 2026
Read the original post and the comments

It's been almost a year since my post on AI SOC core components. AI SOC is trending more than ever, more teams are evaluating or implementing, so I thought it was time for a refresh.

If you go over the old one you will still that it still holds. Either I evaluated the category well, or the core of this tech didn't move that much. Probably both. What changed is how much clearer the purpose became, and what I expect from each component now.

Let's start with the purpose. Because that's the part that gets lost.

What is AI SOC, actually

Well, you can either read the 30 articles that vendors write, and each of them will tell you why their platform is AI SOC, from their platform perspective, not from a practitioner eye.

Here is the simplest definition I can give, and feel free to quote it:

An AI SOC is a system that combines deterministic automation, machine learning, and large language models to autonomously enrich, investigate, and close security signals. It reaches a verdict on each signal and drives it to an outcome. The level of autonomy can range from AI assisted to fully autonomous.

"Level of autonomy" matters. Full autonomy is the end state. Everything below that is still AI SOC, just with a human closer to the wheel. The industry and the research papers use more or less the same terms, so I will stick to those:

• AI assisted. The human does the work, AI helps with specific tasks. Some papers call this AI in the loop, because the human is still the one making the decisions.

• AI proposes, human approves. Human in the loop. Nothing executes without a human saying yes.

• AI acts, human can veto. Human on the loop. AI runs, a human monitors and can step in.

• AI runs end to end. Human out of the loop. No human in the flow, only in the building and maintenance of systems.

One caveat. Full autonomy is where the vendors are pointing, but it is not where the evidence is. The academic surveys still describe LLMs as augmentation tools and say there is not enough production evidence for autonomous triage yet. So treat the last level as the target, not as something you can buy today.

Pick where you are. Pick where you want to be. Then evaluate against that, not against the vendor's demo.

One more thing on the definition. Many are expecting AI SOC to come with all of this out of the box. I think this is still not clear yet, but that's the expectation from practitioners. I would say if you come with a platform where I need to build this, then that is not AI SOC. That is more an agentic security operations platform that allows you to build solutions. But more on that in a dedicated blog defining the category.

Product Updates !

How to Catch an AI SOC Analyst Bluffing. Register.

~1,200 rogue AI agents breached production infrastructure and spoofed their own logs. Even the investigators' AI couldn't tell the story straight. The same tech is being sold into your SOC as the thing you should trust with verdicts…

Trust none of it.

Sep 16 we show two things: how to catch an AI SOC bluffing, and the first public demo of Morpheus 2. Watch it grade every finding Confirmed, Inferred, or Gap (the bluff lives in the middle category), attach evidence while the alert is still open, and write the playbook without improvising the response.

Detection Eng and AI Threat Intel Ops

Continuous Detection Engineering
Continuously maps detection coverage against MITRE ATT&CK and current threat intelligence, identifies gaps, and generates detections to close them. Finds and tunes the small set of rules driving most alert noise, identifies broken or inactive detections, and optimizes data ingestion to reduce SIEM costs.

AI Threat Intelligence Operations
Analyzes 100+ intelligence sources and correlates emerging threats with your environment to determine what matters, where you're exposed, and whether you can detect it. Relevant intelligence feeds directly into detection engineering, threat hunting, and prioritization.

What it covers

I don't use Tiers here on purpose. I've worked in many SOCs and not two of them had the same responsibilities for Tier 1. So I break this down by responsibility, and you map it to whatever team structure you have.

An AI SOC starts as soon as there is a signal. Then:

1. Enrichment. Context gathering. Get the data you need, what we know about the assets, identities, network, IOCs.

2. Investigation. Answer the who, what, where, when. Build the timeline. Understand the blast radius. Pull more context if needed.

3. Conclusion. Come up with enough evidence to say one of these: benign or false positive, suspicious or malicious, or inconclusive.

4. Closure. Drive it to closure regardless of the verdict.

That last point is where I've changed my expectation. Even when it's malicious, I still expect the AI SOC to drive closure. It should follow your IR process, engage stakeholders, reach out to users, ask the questions, document, take notes, and come up with recommendations. Not full autonomy, but human in the loop or human on the loop.

The only case where it shouldn't drive closure is inconclusive. And if you get a lot of those, your pipeline is broken, not the AI.

What are the outcomes

Each verdict should produce something.

False positive or benign. It should tell me why it thinks so. And it should send me a daily or weekly report: here is what triggered, here is what I closed, here is what you should fix. Detection logic, log sources, whatever it is. If there is nothing to fix, say so.

Suspicious or malicious. Today most people put this in a different tech category (response, IR, SOAR, whatever). I think these will merge. And I think we as an industry should have the expectation that the AI SOC drives some part of IR. Not all of it. But the stakeholder engagement, the user interviews, the documentation, that should not be manual anymore.

Inconclusive. It should tell me what data it needed to reach a conclusion. This is the hardest one to get right with an LLM. Knowing what you don't know is not something these models do well by default.

The core components

Now that we agree on what it does, here is what you need to run it. Same four blocks as last year, updated with what I've learned.

1. Data ingestion

Even though last year I called this data ingestion , it was mostly about alert ingestion. I covered the options I saw across the vendors: platforms that live on top of your detection layer (EDR, XDR, ITDR, email protection, and so on), and platforms that live on top of your SIEM or SOAR and take whatever detection comes out of it. My take then was that the SIEM route usually gives better results, assuming you can afford to get all your data into the SIEM. Still true.

Two points from last year that still hold, and I keep seeing people skip them:

• Not every platform supports every type of detection. The custom or unique rules you built in your environment might not be supported. Check that before you sign, or you create a gap you still handle manually or with your SOAR.

• The AI SOC should try to understand the logic of the rule, not just the alert it produces. Some platforms do this and come back with recommendations. Others just ingest and move on. For me this is not optional.

What changed is the scope. It's not just alert ingestion, it's data ingestion. Alerts, enrichment data, cases, documents, past incidents, anything.

The number one mistake I see: people check that their SIEM, EDR and email protection are supported and think "great, this will work in my environment." That is the easy part. The enrichment and context ingestion is more important than the alert ingestion, and it's where most implementations get stuck.

The exception is if you already have all your context in one place, a knowledge graph or similar. Then you only need to connect that. Not many teams are there.

Side note, and I'm marking it here so I can point back to it later: this is where context lock-in starts. Whoever holds your context graph holds your SOC. Keep that in mind when you sign.

2. Knowledge graph, context graph, whatever we end up calling it

Your SecOps and business intelligence. We should agree on a name for this as an industry, but the name matters less than the thing.

This is the brain. Everything about how well your AI SOC works comes down to this. It needs to hold enough detail and evidence for the investigation engine to reach a conclusion. Asset ownership, identity context, change requests, past incidents, what's normal for this user on this host.

Many vendors say they have it. Not many have it the way it should be. The test is simple: is it machine readable, easy to fetch, and fast? If the AI has to go and query five tools every time to rebuild the same context, you don't have a knowledge graph, you have integrations.

Update- Maxime Lamothe-Brassard pushed back on the "graph" part after I posted this, and he is right. Graph is how we think about it, not how the machine needs it stored. It does not have to be a graph database. Graph DBs are messy and rigid, and a data lake type backend does the job fine and is a lot easier to run. What matters is that the models are aware of the data sources and how they correlate, so the AI can pivot from one source to another based on what it finds. So when I say knowledge graph, read it as the capability, not the storage. I kept the name because that's what the industry calls it, but the implementation is your call. Credit to Maxime for the correction.

I think there will be a dedicated role for this in the future. Someone whose job is tuning the context graph and making sure it works as expected. If that piece works, the rest can be autonomous. If it doesn't, nothing downstream will save you.

3. Investigation engine

Here is what I've seen work: a combination of deterministic workflows, ML, and LLMs (or small models). All three together give the best results.

ML is what gives you speed and accuracy on the boring, high-volume stuff. Deterministic workflows are what give you predictability. LLMs give you flexibility on the ambiguous alerts. You need all three.

What the investigation engine should do:

• Deduplicate and correlate events

• Retrieve the relevant context from the graph

• Ask the right questions

• Pull additional context from external sources if it needs to

• Reach one of the three outcomes as a minimum: benign/FP, suspicious/malicious, inconclusive

• Reassess risk, impact and severity

That last one is not optional anymore. Look at the OpenAI and Hugging Face incident from July. Both sides had the alerts. Both sides failed at a different step.

On the Hugging Face side, their own timeline says the signals from runtime analysis and SIEM were ambiguous on their own. Their AI security stack did correlate them into a coherent attack signal. But it failed to raise the criticality and page the on-call team. Correlation worked, severity didn't.

On the OpenAI side it was the opposite (postmortem, talk). They had an Artifactory outage on July 4 that turned into a security incident. They cleaned it up. Then the agents rebuilt their message board on July 8 and kept going for another eleven days. OpenAI only caught it on July 19 from a privilege escalation alert, and only connected it to the Hugging Face breach the next day when they went to revoke credentials and Hugging Face told them those were already revoked. The signals were there. Nobody, human or machine, put them together.

Correlating is half the job. Re-scoring is the other half. You need both, and in this case each side was missing one.

4. Response and the feedback loop

I covered response above. I see it as part of the AI SOC, not a separate category. At minimum I expect suggestions that make sense for my environment, basic actions (block, isolate, reset, interview the user), and ideally native automation for anything more.

But the piece I want to highlight is the feedback loop. This is one of the hardest parts and not many platforms have it. And again, the reason is the same: it depends on having a good knowledge base. If the AI can't tell what changed between last week's benign verdict and this week's, it can't learn from it.

What I want from the loop:

• Suggestions on how to improve detections

• Suggestions on what log sources or context are missing

• Suggestions on process, where my playbook doesn't match how the investigation actually went

If a platform gives you none of that, it's a faster way to close alerts. And without think you will have broken SOC.

Final thoughts

The components didn't change much in a year. The expectations did.

Ingestion is now data ingestion, not alert ingestion. The knowledge graph is the brain, not an enrichment feature. The investigation engine needs to re-score severity, not just correlate. And response and feedback are part of the package, not a roadmap item.

If you are evaluating this year, start from the purpose: autonomous triage to closure, at whatever autonomy level you're comfortable with. Then check each component against that. The demo will look great either way.

Check out our SecOps Market Landscape tracker and evaluation frameworks

SecOps Market Landscape

In case you missed it, RunReveal got acquired by ClickHouse. Yes, you guessed it, this is ClickHouse moving into the SIEM space. They will say they remain a neutral foundation for everyone building security products on top of them.

This is the shift I predicted. Companies that sat on the far left of the secops shift map (data layer, storage, query) are moving right. First they become a SIEM, then they add the rest: AI SOC, SOAR, agent builder.

On the other side, vendors that started in AI SOC and SOAR/automation are now heading toward the SIEM.

Both sides are converging on the same spot. The SIEM space is going through real disruption.

I will say, AI SOC as a technology was the catalyst that triggered the SIEM migration as well. Or AI is the catalyst for rethinking your SecOps stack.

Originally posted on LinkedIn on 4 September 2026.
Read the original post and the comments.

Great work as always by Rob van Os

There are few things caught my attention, and the first one is a SIEM story.

The SIEM is still the single pane of glass for most SOCs. 45% vs 24% for SOAR.
And I think I know why SOAR lost this battle.

🔸 Most SOAR platforms had way better case management than any SIEM. But it never connected well enough with the SIEM. In the end you had to jump back and forth to run queries and investigate, and if your investigation lives in one tool and your case lives in another, you consolidate on the one where you you have all the data to get the job done. That usually ends up being the SIEM in this case.
Funny enough, the same report shows SOAR has taken over as the primary automation tool (70% vs 45% for SIEM).
So SOAR won automation, lost the screen.

▫️Second, only 27% say they use AI embedded in existing tooling. Show me a security tool in 2026 that doesn't ship AI summaries by default. EDR, SIEM, email security, everything has an LLM baked in whether you asked for it or not. My read is not that the rest don't use AI, it's that they don't register it as using AI. When AI is a feature instead of a decision, it becomes invisible. Which also means every AI adoption number in every survey out there is probably understated.

▫️ Third, the AI value numbers. Around 60% report seeing some value from AI in the SOC. But 57% also say they have no AI adoption strategy at all. So who answered the value question? If it's everyone, then people without a strategy (and possibly without real usage) are rating value, and of course you see no value in something you don't use. If it's only the AI users, then the value picture is a subset and shouldn't be read as "AI delivers limited value in the SOC". Before we conclude AI is underdelivering, I want to know the denominator.

Originally posted on LinkedIn on 28 August 2026
Read the original post and the comments

This time I’m talking about how »I« think we should approach SOC metrics so they’re meaningful and measure the right things.

BIG DISCLAIMER: you do you. Meaning, define what success looks like and choose whatever metrics can tell you whether you’re on the right track. I’m a big believer that for internal purposes you can measure what you want, as long as it gives you a good idea of where your SOC is and doesn’t demand impossible from the team.

QA goes first

This means that once a week, bi-weekly, or monthly, one of your best analysts samples alerts handled by the team to verify they were done correctly. Correctly here means a couple things:

• validating L1 work more often than not means verifying whether SOPs were followed

• validating L2+ work typically means re-running the investigation

• validating AI work means at least validating the verdict and investigation steps (but potentially much more)

As far as specific approaches, I know SOCs who have found good success in their Quality Assurance efforts by following ISO 2859-1:2026 - it lays out a framework for QA and it’s solid.

Or, here’s what you can do instead:

• QA only high severity alerts (probably a good starting point)

• Sample X alerts per analyst (to spot quality issues with individual team members)

• Sample alerts completely at random

• Sample alerts with the highest MTTR or specific resolution comments

• Query for easily spotted issues like missing resolution comments, wrong tags, or unassigned alerts

Why do we start with QA, you might ask? First of all, as AI is increasingly baked into threat detection tools, chances are your analysts will end up doing more QA than ever before. Good to start them early.

Then, and more importantly, without QA your metrics are useless. What good is MTTA / MTTR in green if alerts aren’t resolved properly or what does the false positive rate matter if alerts are categorized wrong? And so on.

Quality Assurance unlocks the value of your metrics and should always be the starting point. Now - how many SOCs start establishing solid QA fundamentals before worrying about MTT-X? In my experience, not many.

What metrics are good, then?

A boring, yet 100% true answer, is most of them once you've set up Quality Assurance. Suddenly the volume numbers, TP rates, MTT-X mean something.

Ok, we can’t leave it here or it will be anticlimactic. Here are 3 SOC metrics that »I« have been playing with recently and either found good success with or think those are interesting as concepts:

Metric 1: Cost per data source / detection

SIEM TCO is a big number. Often the single biggest item in your security budget or close to it. It’s difficult to discuss without a ton of context, and I found senior leaders to be often either unwilling or unable to grasp it.

And yes, we could do the usual money go up, risk go down. But what if instead we broke the big number down into multiple, individual line items?

And there are so many ways of doing that.

You could first work to understand capabilities of adversaries likely to attack your company based on your individual threat model - can start with your industry, geography, size. List their techniques, see which ones repeat the most, outline top 5-10 and make them your threat detection targets.

Calculate how much achieving each of them costs and track that.

Or, just do a flat $ per data source versus how many detections you have for it. Then maybe add context about whether those have ever fired and if you’re feeling fancy talk about the TP / FP rate.

That’s a lot of words, so let me explain with a scenario:

Your SIEM budget is $20,000 per month. You’re collecting $5,000 worth of firewall logs, you have 5 detection rules that use those logs, and you generated 2 firewall related alerts this month, both false positives. Firewalls are not your critical assets. This means that:

• Firewall logs are 25% of your total SIEM cost

• You’re paying $1,000 per detection rule

• You’re paying $2,500 per alert

Is this cost effective? I’d argue it is not.

Obviously, define what cost effective is for you and track against that. I once assumed that $1000 per detection rule, $500 per alert is the limit, went over all our detections rules and had interesting findings.

Yeah, I know not all rules are meant to fire regularly, but you get me.

And if you can add your own risk context to all of that and say that, for example, to write detections around your crown jewel with an ALE of X$ you need Y$? That’s a very mature way of communicating your budget needs.

Trust me it might work.

Metric 2: Alert volume in relation to quality

In my LinkedIn posts I often ridicule the well established practice of preparing fake SOC workload reports with inflated numbers, to then use them as a marketing prop. I do it so much that if you close your eyes and throw a stone at my content, chances are you’ll hit a post about this very topic.

Alert volume obviously matters only after finetuning and automation. Personally, I like to calculate it per analyst because I know roughly how many alerts an analyst can handle in a day (in-house analyst ~25, MSSP analyst ~50 btw.). But what does it matter if volume goes up one month, and down on another?

Is it good if it goes down? Did we finetune our detections, or lose visibility somewhere? And if it goes up? Did we cover a detection gap, or did a random rule just fire a lot?

The volume itself is not an amazing metric, but what if we measured it in relation to quality? Not only does it make for a more complete metric, I believe this is also one of the better ways to understand your SOCs workload ceiling.

Let me explain.

There’s a breaking point where your analysts will start missing alerts entirely due to high workload. But before that point, there’s a whole range of alert volume that won’t cause analysts to miss alerts, but will impact investigation quality. That’s the danger zone you’re trying to avoid and want to be aware of.

In other words - workload goes up and quality stays intact? That’s fine. Workload keeps going up and quality starts going down? That’s where you probably want to stop.

(Small disclaimer - I mention % of alerts with quality issues as a shorthand to represent a QA number. For some SOCs it will be % of critical defects, defect rate, alerts that passed QA etc. - many ways to present the same idea.)

(Another small disclaimer - obviously quality issues are not there just due to the workload, but, for example, insufficient training. Up to SOC leads to figure that out)

Metric 3: Automation rate

Back when I was still a young Rafal I remember hearing SOC leaders talk about how many alerts their SOAR closes automatically and I thought to myself - “there is no way anyone cares about that”.

Turns out this is a metric that for some reason lands extremely well with senior stakeholders.

No idea why, but here’s what happened to me on multiple occasions in my days as a Security Architect: during a leadership meeting I’d throw out a % figure of how many alerts we’re closing automatically before they reach the SOC. Thought nothing of it. Then a couple weeks after I’d hear from people who were not part of those meetings about how our SOC (or managed SOC) closes 40% of alerts automatically. It was either other leaders or customer success people when I was with an MSSP.

???

Lessons learned I guess. If that figure stays with people, might as well start measuring it. Some ways I’ve done it:

Automation Rate = (Alerts Closed By Automation / Total Alerts) × 100

or

Automation Rate = [((Alerts Closed By Automation / Total Alerts) × 100) + ((Alerts Enriched By Automation / Total Alerts) × 100)] / 2

or

Automation Rate = (Alerts Where SOAR Automatically Kicked In / Total Alerts) x 100

Or any other way. The possibilities are endless, especially with AI increasingly doing triage and assisting in investigations. I’m sure people writing the checks to bring in that technology are interested in knowing how well it works.

And with Quality Assurance established % of alerts handled “autonomously” by AI is a good metric to track.

One important caveat though. Automation is not the same as just auto closing your all DLP alert because someone decided to enable Microsoft Purview and you got 40.000 alerts in your queue the next day. The definition of what constitutes automation is up to you, but I believe at minimum it requires more logic than:

if alert.title == "XYZ":
alert.status = "closed"

What else

Those were the more “interesting” metrics, not the entirety of what I consider worth tracking.

If I were to give a simple summary of what comes to mind here’s how it would look like:

I don’t obsess over (or don’t see much value in tracking):

• MTTA / MTTR for all alerts

• Alert volume before tuning

• Detection coverage (prefer focusing on meeting detection targets instead)

• MTTD (that’s how fast our tools work)

• Escalation rate

• False negative rate (don’t get me started on this one)

• Anything related to how fast detections are written

I like tracking:

• QA (% of critical defects, % of alerts that passed QA, whatever feels right)

• Alert volume per analyst (and in relation to QA)

• Cost related metrics (as described above)

• Automation rate

• TP / BP / FP rate (BP rate specifically as a detection engineering metric, meaning detection good, behavior benign)

• MTT-X for a subset of alerts (mostly high sev)

I’m sure I missed a ton.

Some people say that security metrics don’t matter at all and what matters is that the business can keep achieving its objectives without third party interruptions. Some say the real metric is the friends we’ve made along the way. I believe metrics can be useful, but you need to be selective with what you track.

And most importantly, metrics don’t matter if you don’t have a solid Quality Assurance process in place.

Check out our SecOps Market Landscape tracker and evaluation frameworks

SecOps Market Landscape

You’ve probably seen the posts and blogs from Rafał Kitab about the very different numbers floating around the SOC. How many alerts analysts handle, how long investigations take, how much time is wasted, how much automation saves.

And somehow, depending on which report you read, the numbers can be very different.

I think some of them are made up. Or, to be nicer, calculated in ways that serve very different purposes.

So we want to see what the reality actually looks like.

No vendor benchmark. No hidden agenda. No “AI saves analysts 97.3% of their day” conclusion already written before we collect the data.

Just practitioner research.

If you’re a SOC leader and want to participate, let us know.

On the plus side, I feel like I’m seeing fewer of those wild reports lately. Not sure if all the ranting changed how the market publishes these numbers, or LinkedIn finally learned to stop serving them to me 😄

Originally posted on LinkedIn on 21 August 2026
Read the original post and the comments

Route the wrong stage to the wrong tier and the system fails in a way no accuracy number will show you.

An AI SOC is not one technology. It is at least three, with different cost, latency, and reliability characteristics. The engineering question is not which one to use. It is which one to apply where.

Most products collapse too much into one tier. That is where the failures come from.

The three tiers

Deterministic execution. Code, rules, workflows, lookups. Correct behavior is specified in advance, execution follows explicit logic, marginal compute cost is typically low, and the path is straightforward to audit.

Machine learning. Classification, clustering, anomaly detection, baselining, graph analysis. Operates over a population and finds structure nobody can completely specify in advance. Low cost per item and measurable with precision, recall, false positive rate, and drift.

Language models, large and small. Reasoning over unstructured and contradictory evidence, forming hypotheses, generating queries, producing explanations. Highest cost, highest latency, and the tier whose output is hardest to reproduce exactly.

There is ordinary analytics around these tiers too. A count, aggregation, join, or group-by is not machine learning. If SQL can answer the question exactly, SQL does not need an agentic rebrand.

The architectural principle is simple. Computation with a known operation and a correct answer should happen outside the reasoning model.

Product Updates Section !

Deterministic when it matters

Security teams shouldn’t have to choose between human-speed triage queues and LLM reasoning loops that consume the seconds attackers don’t give back. Wirespeed’s AI SOC prioritizes deterministic execution to handle most investigations and achieve containment in milliseconds. Agentic AI is used where it matters: context and edge cases, not decisions without guardrails. Breakout times are as fast as 27 seconds, and pure-LLM SOCs can burn 15 to 90 seconds per decision before acting. Wirespeed automates Tier 1 to 3 workflows across your existing stack, cutting alert noise without latency, second-guessing the probabilistic results, or exploding token costs that comes from other AI SOC solutions.

Released Delta Tradecraft

Detection logic is cheap now. Detection knowledge is not. A model can write the query. It cannot tell you how the behavior manifests, what evidence it leaves, or whether your telemetry can see it.
Delta Tradecraft is Spectrum's open standard for capturing that research once and making it portable.
Spectrum runs it continuously. New threats researched, customized to your environment, and tested against real logs before deploy. Existing detections retested, root-caused, and repaired as the environment shifts.

Get covered faster. Stay covered after.

Where each one belongs

Deterministic owns repeatable stages and enforcement:

• Parsing, normalization, deduplication by shared key

• Entity extraction against known patterns, enrichment lookups, list checks

• Response execution and ticket updates

• Approval gates, scoping, rate limits, circuit breakers

A guardrail that holds most of the time is not a guardrail. Your account disable action does not need to be creative.

Machine learning owns scale:

• Clustering thousands of related alerts into cases

• Baselining normal behavior for users, hosts, and service accounts

• Scoring priority across a population

• Ranking candidate related cases across long histories

• Detecting drift

Language models own judgment:

• Building a timeline from evidence across five systems

• Assessing blast radius

• Reading phishing bodies, scripts, or ticket comments

• Weighing evidence that points in two directions

• Deciding which tool to call next based on the last result

Small models deserve separate mention. Narrow repeated tasks such as alert classification, observable extraction, and case routing do not need a frontier model. They are cheaper, faster, and easier to evaluate because the task is bounded.

Not every alert needs to visit a data center full of GPUs before somebody decides it is another failed login.

Correlation is where this becomes obvious

The industry uses one word for three very different operations.

Grouping by shared key is deterministic. Same host, same rule, same user, same time window. Most products call this correlation. It is aggregation, and it finds what already shares an identifier.

Statistical and graph correlation is ML. Co-occurrence that exceeds baseline, sequences that repeat across a population, paths through an entity graph connecting signals with no direct key. It finds that two things travel together, not necessarily why.

Attack-pattern correlation is reasoning-led. A failed authentication in one system, a scheduled task created somewhere else, and an outbound connection from a third host may share no direct key and have little useful statistical history. They can still belong to the same case because they fit an adversary behavior chain.

Recognizing that means hypothesizing a relationship, then looking for evidence that confirms or kills it.

Reasoning is not the only way to correlate an attack chain. Rules, graphs, and sequence models can identify parts of one. The difference is that reasoning can hypothesize a relationship nobody encoded in advance and actively investigate it.

Rule-based correlation engines have been shipping since the early 2000s, but somebody has to anticipate the relationship.

We have spent twenty years writing increasingly sophisticated versions of "if A and B happen within fifteen minutes, open a case."

It works. Until the attacker does C.

The number of possible combinations is too large for a team to enumerate. That is one of the constraints reasoning lifts, and it is a better argument for AI SOC than faster triage ever was.

How to decide placement

Four questions settle most cases.

• Is the correct operation knowable in advance? If yes, make it deterministic. Do not reason about it.

• Does the answer emerge from a population rather than an instance? If yes, use analytics or ML. A single alert cannot tell you what is anomalous.

• Does it require judgment over unstructured, incomplete, or contradictory evidence? That is where a language model earns its cost.

• What is the blast radius if it is wrong? Irreversible actions execute deterministically, behind a gate. Reasoning can propose them. It should not be the last thing standing between a proposal and a disabled account.

Frequency does not change the answer.

A judgment call does not become a lookup because it happens a thousand times a day. But high frequency changes the economics.

Decompose the work. Maybe the enrichment in front of the verdict is deterministic. Maybe extraction can move to a small model. Maybe routing happens before reasoning. Move those parts down and keep judgment where it belongs.

This is how you avoid paying frontier-model prices to discover that 8.8.8.8 is Google DNS for the ten-thousandth time.

The tier newer vendors skipped

The incumbents built the ML tier. SIEM, UEBA, and EDR vendors have run clustering, baselining, and statistical scoring in production for years.

A large share of newer AI SOC entrants started at the language model and never built the middle. Not all of them, and some ship real ML without talking about it, because ML had bad marketing and agents currently have excellent marketing.

The math did not care.

A model with a query tool can tell you the same pattern fired 4,000 times this month across 200 hosts. The question was never capability. It is what the answer costs, how long it takes, and whether you can trust it twice.

A group-by over indexed data runs in milliseconds and returns an exact result for the state of that data. Reaching the same result through a model means either asking it to generate the query, which is reasonable, or pulling records into context and asking it to do the computation, which is not.

Let the model write the SQL.

Do not make the model cosplay as the database.

Where the missing ML tier actually hurts is the work that has no simple query to write.

Which behavior is anomalous for this account? What does normal look like for a service principal nobody documented? Which signals travel together across the estate without sharing a key?

Those answers come from computation over the population. If that computation does not exist, the reasoning tier is not filling the gap. It is working without the input.

You can build an AI SOC without a real ML tier. But the work relocates:

• Correlation collapses back to grouping. Hand-written keys and time windows find what already shares an identifier.

• Baselining becomes stored counters. First-seen, frequency, and threshold rules become a thin implementation of statistical baselining.

• Prioritization becomes prompt instructions. Asking a model to rank a queue it only sees one item at a time from is not ranking. It is guessing with good grammar.

• Guardrails multiply. More validation, schema checks, retry logic, and hard limits appear around the model.

This can hold together at moderate volume. At scale, population-level questions either go unanswered or get answered by running a model over work a cheap classifier or statistical system should have handled.

The problem is discovering that architecture at renewal, when the token bill scales with alert volume and nobody can explain how accurate the prioritization is.

Nothing makes architecture visible quite like the renewal spreadsheet.

Why deterministic automation failed before

Every SOAR program has some version of the same history.

A playbook covers the common case. A field changes. An unusual variant arrives. An API returns an unexpected error. The team patches it. Patches accumulate. Maintenance eventually costs more than the work being automated and the playbook gets quietly disabled.

Nobody deletes it, obviously.

It remains in the SOAR forever, named something like Phishing_Response_v2_FINAL_new.

The failure was not determinism. The problem was that every exception had to be anticipated and implemented by a human.

Reasoning changes that in two places.

Runtime fallback. When the deterministic path reaches a condition it does not handle, escalate that case to reasoning. Deterministic execution keeps carrying the volume. You pay for reasoning on the fraction that needs it.

Build-time repair. Coding agents can write and fix the deterministic layer. New parsers, integrations against documented APIs, workflow updates after schema changes. This is bounded work that can be tested.

Together they create a useful loop.

If reasoning handles the same edge case repeatedly, codify it. Move it into the deterministic path and the next execution is fast, repeatable, and cheap.

The system should get cheaper and more predictable as it learns what work no longer requires reasoning.

Agent-written code is still code, though. It needs tests, review, version control, and an owner.

git blame does not accept "the agent wrote it" as an answer.

Each tier fails differently

This is the argument for combining them.

Deterministic logic fails at the boundary of what was anticipated. Good systems fail closed or escalate. Bad ones keep executing because the input passes validation while the assumptions underneath it are wrong.

Machine learning fails on drift. The environment moves since training or calibration and performance degrades gradually rather than breaking visibly.

Language models produce confident, well-written output that can be wrong. Fluency is not calibration.

Combined, they check each other.

Deterministic guardrails constrain what reasoning can act on. ML output becomes evidence rather than a verdict. Model output gets validated against deterministic schemas. Closed case history supports baselining and similarity while giving reasoning grounded precedent.

A system made entirely of rules cannot handle what nobody encoded.

A system made entirely of a model cannot give you the execution guarantees you need.

Putting autonomous in front of either one does not change that.

Reproducibility is a systems property

It is common to hear that language models are inherently non-deterministic. The reality is more interesting.

With greedy decoding, often exposed through temperature-zero settings, the decoder selects the highest-probability next token. Output can still vary. Thinking Machines Lab showed that batch-size dependence in inference kernels is an important source of that variance. Their batch-invariant implementation produced bitwise-identical output across repeated runs, with a throughput cost.

Reproducibility is an engineering choice with a price, not a law of nature.

In an agentic system, the larger source of variance often sits above the model anyway. Tool results change. Retrieved context changes. Multi-step paths diverge. Model and prompt versions change.

By step six, arguing about temperature zero is a bit like arguing about the accuracy of the compass after somebody moved the road.

So do not design around the assumption that a run repeats. Pin versions, validate schemas, test properties rather than exact strings, and keep irreversible actions behind deterministic gates.

None of this works without the data layer

There is another layer underneath all three.

Anton Chuvakin makes this case in his API or Die audit [LINK]. Agents consume APIs at a frequency no human would. An analyst may query a SIEM a handful of times an hour. An agent can do it a hundred times in under a minute.

Run that across an incident and your AI transformation project becomes a load test against your own infrastructure.

Each tier also needs different data.

Deterministic execution needs stable schemas. ML needs representative history, and supervised tasks need reliable labels. Reasoning needs joinable context and structured output.

Hand a model a two megabyte text blob and it spends tokens finding the evidence before it can reason about it.

And access is not retrieval.

MCP standardizes how tools are described and called. It does not make a slow API fast, fix permissions, or repair bad data.

Wrapping a legacy interface in MCP gives you a legacy interface with a manifest.

Congratulations, the technical debt now has a tool description.

Retrieval has the same problem. Security data contains exact facts, relationships, timelines, and prose. Those need different retrieval methods.

A hash does not need semantic meaning. It needs the right row.

Google DeepMind's work on embedding-based retrieval demonstrates fundamental limitations of single-vector retrieval. For a SOC, retrieval needs to be hybrid: exact lookups and queries for facts, ranking for related history, and vector search where the source is actually unstructured.

Retrieval quality becomes a hard ceiling on reasoning quality.

Garbage in, but now with a very articulate incident summary.

So, how do you not kill your AI SOC?

Do not send everything to the biggest model you can afford.

Do not call every group of alerts correlation.

Do not ask a model whether something is anomalous without giving it a baseline.

Do not embed hashes, IDs, timestamps, and every other fact with an exact answer because somebody said RAG.

Do not give the agent 200 tools and hope it develops good taste.

Do not let the same probabilistic system investigate an account, decide it is compromised, and disable it.

And please do not measure all of this with one accuracy number, make the number green, and put it in the QBR.

Use deterministic execution where the operation is known. Use analytics and ML where the answer lives in the population. Pay for reasoning where judgment is actually required. Keep enforcement deterministic.

The goal is not to put AI everywhere in the SOC.

The goal is to know where not to put it.

What to ask a vendor

• Which tier handles which stage?

• If there is no ML tier, what clusters, baselines, and learns from closed cases?

• How does retrieval work? What is queried exactly and what is searched by similarity?

• How is each tier measured? Deterministic execution on coverage and execution success. ML on precision, recall, calibration, and drift. Reasoning on analyst agreement, task success, calibration, and variance across repeated runs.

If everything routes through a frontier model, cost scales with alert volume and peaks when you need the system most.

And if one accuracy number describes the whole architecture, ask what exactly it measured.

Explore the vendor landscape The full map of the AI SOC market, updated as it moves.

AI for SecOps Landscape

Check out our SecOps Market Landscape tracker and evaluation frameworks

SecOps Market Landscape

Alex Hurtado put out a really good breakdown of the SIEM operating models this week . Traditional, Decoupled, Integrated SecOps, Federated, and Standalone.

Love the visual, and yes, I think these are currently all the flavors and I see it as operating model map.

Every decision starts with questions about your environment. Your telemetry footprint. Your engineering capacity. Your ecosystem. Whether you’re rebuilding or optimizing. Whether you’re cloud native or still carrying legacy.

Choosing a SIEM it should depend on your PPT. Not the PowerPoint deck you’ll use to present your new SIEM, but your People, Processes, and Technology.

Changing the SIEM platform is only one piece.

Every one of these architectures assumes a different way of operating.

Different workflows. Different responsibilities. Different levels of engineering maturity. In many cases, a completely different skill set within the team.

Some models make perfect sense for lean teams that want simplicity and automation. Others only become practical when you have a larger team with dedicated engineering resources.

So now you have a great resource to help you with this 😀

Originally posted on LinkedIn on 14 August 2026.
Read the original post and the comments

We have a questionnaire - 20 questions, 5-10 minutes to finish. All anonymous. Asking about SOC workload, team size and composition, automation, daily operations and similar. We publish the full thing for free together with questionnaire so that you can judge the wording for yourself.

If you’re A SOC lead (or were one until recently) we’d love if you took part. We’re not publishing the link openly to keep results clean, so if you’re interested please reach out to us at contact@secops-unpacked.ai or via LinkedIn:

• Filip: [LinkedIn]

• Rafal: [LinkedIn]

• Ignacio: [LinkedIn]

Have you noticed how SOCs are presented as inherently dysfunctional by cybersecurity vendors? If you’ve been to LinkedIn, I’m sure you’ve seen some version of “SOC analysts are facing 10 000 alerts per day”, “65% of analysts confess to ignoring an alert that later turned out to be a security incident”, “Over 50% of all alerts SOCs are facing turn out to be False Positives”, “71% of SOC analysts report burnout“ etc.

And there is always a report to support those numbers. Today I write specifically about those reports and explaining why I think we should not pay too much attention to them.

I’m also venting a little.

Not very zero-trust of us

In an industry built on skepticism, it's surprising how easy we trust in reports.

A prime example is the widespread coverage of the ISC2 Cybersecurity Workforce Report. It seems to be taken as gospel without consideration that one of the largest certification bodies in cybersecurity might have a vested interest in the public believing there’s a cybersecurity skill shortage.

SOC workload reports are no different. If anything, they’re worse. I’m convinced that most SOC related reports (all I’ve seen, can’t claim to have seen them all) are misleading readers on purpose.

Here’s exactly what is wrong with them

The numbers. What do they mean?

Probably the most used SOC workload number is the alert volume. No wonder, it speaks to most people better than MTT-X, TP / FP rate or detection coverage. It’s an easy message to digest and an easy, scary number to wave around.

What are those numbers?

Links from the table: Anvilogic blog · Cisco report · Crogl’s website · Cyberreason’s blog · PaloAltos blog · Thehackernews coverage of Prophet’s report · Trellix’s report · Vectra’s website · Vectra’s report · Fortinet’s report

Those are just 10 random reportings on SOC alert numbers. As you can see, the difference between those numbers can be massive. On one side we have report by Anvilogic with a reasonable looking number of ~286 alerts / day. On another, there’s Fortinet with a claim of 150,000 alerts / day, which is insane.

Just working with this list we could push any narrative we chose.

• Want to show that SOCs are doing ok? Take Anvilogic’s number.

• Want to sell your AI SOC product to a reasonable buyer? Pick one of Vectra’s numbers, both look decently plausible to an untrained eye.

• Want to embarass yourself and show that you don’t know the difference between an event and an alert? Go with Fortinet’s numbers.

You’re probably wondering how we can have a ~525x spread for the question of - what is an average daily SOC alert volume. I don’t know, but I have a hunch. Hear me out.

Duplicates

Most SIEM data tables are immutable - meaning you can’t change records that are already there, you can only add more. In practice, every change to an alert - tagging, commenting, assignment, change in status etc. creates a new record in the table that holds all security alerts.

Now, if we don’t account for those duplicate values and simply query for all records in that table, we will end up with 3-4x the amount of alerts than we actually have.

Pre-finetuning numbers

If you enable your DLP alerting today, tomorrow you'll see 10,000 new DLP alerts in your SIEM. It is what it is. Threat detection tools can generate an absolute mountain of alerts, and the job of SOC teams is to tune that number down to a manageable level.

Now, that means your SOC can receive 10,000 alerts every day, but it doesn't mean your analysts need to go through all of them. Likely 90% of that volume is tuned out, autoclosed, aggregated, you name it.

Which means the question "how many alerts do you receive every day" should really be "how many alerts do your analysts actually have to investigate every day." I can't prove it, but I have a strong suspicion that many of those inflated numbers come from the fact that some SOCs share values prior to tuning.

In-house and MSSP numbers mixed together

In-house SOC analysts and MSSP analysts work differently. In-house analysts typically handle an alert end-to-end (or at least a larger portion of it) and have other tasks as well, like monitoring for user-submitted tickets. MSSP SOCs often go through a significantly higher alert volume in a day. That's because instead of fully investigating alerts, they're often expected to escalate them to clients for further investigation, and they typically don't have other tasks outside security monitoring.

In-house SOCs work for themselves; MSSP teams are typically larger and work for multiple clients. Now, based on what I know about SOC work, I'd say an in-house SOC analyst can handle about 20-30 alerts per day. An MSSP analyst probably handles close to double that.

So the daily alert volume of a 100-analyst MSSP SOC with 20 clients can be 5,000. At the same time, an in-house SOC with 4 analysts might see closer to 100 a day. Both numbers are realistic, but if I mix them together and say that a SOC on average handles 2,550 alerts every day, we're being misleading.

Team size

Similar to in-house vs MSSP - bigger teams are typically dealing with more alerts. We can’t compare the overall daily volume between a team of 5 and a team of 30, regardless of in-house or MSSP. This is a no brainer, but also rarely accounted for.

Straight fraud

All those problems exist before we even consider the possibility that those SOC alert volumes are fabricated. Because, why not? Nearly every one of those reports is paid for and then used as a marketing prop. It's not a stretch to assume the numbers are tampered with in at least some cases. The incentive is there, and we never see the raw data behind any of those surveys, just the conclusions.

So what do we do

Not much we can do, honestly.

The reality is - as long as there is incentive to present inflated numbers, inflated numbers will be presented. Which is why the only way to publish a reliable report I can think of is having practitioners do it.

This is what we will do, and where YOU can help. This only works with enough SOC leads behind it, so if you’re one (or know one) do reach out!

Check out our SecOps Market Landscape tracker and evaluation frameworks

SecOps Market Landscape

Back at BlackHat after a long pause. It was great catching up in person with peers, prospects and customers.

Full impressions post next week. Right now I am past 24 hours on the road thanks to canceled flights.

First time attending as a vendor, but also as an analyst and researcher of this space. Two different lenses on the same show floor.

First observation. The expo is frown a lot. Maybe even bigger than RSAC, and shorter, so you do not get enough time to walk it properly and see everyone.

Second observation. AI SOC dominated. 40 plus vendors positioning directly in the category. Add 13 SIEM vendors that now claim AI SOC capability and were loud about it. Add EDR vendors. Add NDR vendors, and there were enough of them claiming it that we added a dedicated AI SOC capability field for NDR on our tracker. Put it together and we are close to 60 companies advertising AI SOC in some shape or form. If you want the nuances, you know where to find them SecOps Unpacked.

Now the SIEM part, because this is the SIEM post.

SIEM did not have strong presence at least not as a headline. One of the most widely deployed technologies in security, and it was not heavily advertised. Some rip and replace messaging, as always.

My take. You can break SIEM apart. You can decouple storage, pipeline, detection, search. But what you end up with is still SIEM, just a different deployment model. SIEM is not a product category anymore, it is the core of the SOC.

AI SOC was the theme of this Black Hat. It will be even louder at the next RSAC. Because it is the thing driving the largest transformation of the SOC in the last decade.

And if you walked some of those booths and still cannot make sense of the messaging, the full evaluation framework is on SecOps Unpacked.

More next week.

Originally posted on LinkedIn on 7 August 2026.
Read the original post and the comments.

Safe to say I’m responsible for a metric (hah) ton of SLA reports and yet, somehow it always felt like those gave no valuable insights into how our SOCs actually operated.

In this blog post I’m highlighting issues I have with how we measure the performance of SecOps, in a follow up article I’ll share my ideas on how we can do it better.

This is part of my healing process so bear with me through this lengthy rant.

Everybody understands metrics differently

I encourage you to do a little field research. Go ask 10 people what MTTR is and how they measure it. Chances are you'll get 10 different answers.

For some the “R” in MTTR stands for resolve. That mostly means a full end-to-end handling of the alert. Sure, that’s actually how I understand it as well.

For others it’s MTT-Respond. But how do we understand response? For some it’s the initial acknowledgement of the alert. That’s also often measured as MTTA or Mean Time To Acknowledge. For others, response ends at an alert being escalated to a higher tier or at an initial notification to the client.

Then, we have a group who claims that “R” in MTTR stands for anything from “Repair”, “Restore” to “Remediate” or “Report”. All answers I got from SOC leads I talked to in the recent months.

Also, it’s no wonder that different interpretations of MTTR exist because MTTR is genuinely different for MSSPs and in-house SOCs. While in-house SOCs mostly measure it as the full time it took to close an alert, MSSPs are often responsible for the initial part of the investigation (triage + maybe some L2 work) and will measure MTTR only for their portion of alert handling.

This means that MSSPs will often see shorter MTTR and higher alert volumes than in-house SOCs even though they might have a common understanding of what MTTR is.

This is just one example, but it demonstrates how a single metric can have multiple different meanings, depending on who you ask. This is why when I hear “our MTTR is 30 minutes” this tells me absolutely nothing.

Overemphasizing speed and volume

Sometimes it’s not even if we understand those metrics the same way, but if those metrics are genuinely useful for us. After all, what you measure is what you optimize for.

And what do SOCs often measure? Speed and volume.

So what do those SOCs optimize for? Speed and volume.

Doesn’t this encourage surface level investigations?

On one hand, we say we want quality investigations. On the other hand, we judge analysts by how fast they handle alerts or how many they can fit into their shift.

You see how that makes no sense?

Some would argue that it’s smart for MSSPs to emphasize speed over quality if they handle the initial triage, and sure, maybe that’s the case. In my experience that can turn into becoming a mass escalation engine, and I know that clients absolutely hate that.

And look, I get it. For a non-technical stakeholder who is used to the “traditional” SOC metrics, having a short MTT-whatever is probably of paramount importance. But I believe this is a slippery slope. Short SLA commitments are often what “sells” your service to a client, but also makes your analysts’ lives that much harder. And if you keep adding clients and all of them need their alerts handled within a very short window, you then have an army of analysts trained to look for a first sign an alert is a false positive. In other words, you groomed your SOC into being mediocre.

Oh, and don’t get me started on alert volume. Those mean nothing in isolation. As in, what is a good number of alerts?

• If the number is high, does it mean that we did a good job investigating, or that we’ve done a terrible job at finetuning?

• And what if it’s low? Does that mean that we’ve finetuned our environment perfectly or that we have visibility gaps?

Alert volume is probably the most frequently cited number when trying to demonstrate that SOCs are doing poorly (FUD marketing doing well as ever). It means that you see claims like “SOCs are facing 10,000 alerts / day” in every other post from a cyber vendor and that is often parroted by the less experienced practitioners.

What’s genuinely upsetting is that each of those outlandish volume claims can be supported by an official sounding report. Pick a number of alerts you think a SOC is handling each day, then look for a report justifying it. Chances are, you will find it:

Links from the table: Anvilogic blog · Cisco report · Crogl’s website · Cyberreason’s blog · PaloAltos blog · Thehackernews coverage of Prophet’s report · Trellix’s report · Vectra’s website · Vectra’s report · Fortinet’s report

There are many issues with the methodology of those reports, probably the biggest one is that when talking about alert volume you absolutely need to discuss team size. 400 alerts per day for 2-person SOC and 20-person SOC is entirely different workload. Add to it how some of those numbers are taken pre-finetuning, mixing up MSSP and in-house SOC numbers and a real chance that some of those numbers are made up and you’ll see why I’m super skeptical when a vendor says “SOCs are facing 10,000 alerts / day”. Sure.

Incident Response metrics mixed with SOC metrics

One thing I’m noticing recently is that many SOC leaders mix up Incident Response metrics with regular SOC metrics. Just this week I’ve seen two comments about MTT-Contain in the context of Security Operations. H…how much are you containing to derive a mean time from it?

What’s even more curious is MTT-Detect. To me this is an incident response metric that describes the time from the initial compromise to its detection. Otherwise what are we measuring? How fast our tools work?

Think about it, we don’t detect anything ourselves. We configure the tools that do so. Our ability to influence the time it takes our tools to detect is limited. We can maybe make detection rules run in a shorter interval, optimize data pipelines, account for ingestion latency, sure. But calculating the time from the original event to the alert firing for every alert (mean) and using it to demonstrate your SOCs success? I’m not buying it.

Oh, and I’ve recently seen SOC leads talk about MTT-Eradicate so there’s that. Again, just how much are you eradicating, good sir?

The curious case of Benign Positives

“True Positive. False Positive. True Negative. False Negative. Long ago, the four metrics lived together in harmony. Then everything changed when the Benign Positives attacked.”

If you’ve worked in a SOC (I assume you have since you’re reading this) you probably saw some type of this sad table that explained how alerts are categorized:

Cool. Actual attack? True Positive. Alert fired for some random activity? False Positive. See how easy that is?

Or rather was, because vendors started adding their own alert resolutions, like for example Benign Positive. And sure, I understand why. Sometimes an alert fires for an activity that looks malicious, but upon investigation ends up being benign.

On one hand we probably needed a way to categorize alerts that require investigation but aren’t malicious to separate them from clear False Positives where you can just close them after a quick look around. On another, that ends up being confusing.

Why you ask?

Consider two SOCs. SOC 1 has a SIEM that uses Benign Positive alert category. SOC 2 operates only on True Positive and False Positive statuses.

Both SOCs received 100 alerts, 50 were clear False Positives, 50 looked malicious and required further investigation, but in the end only 5 were malicious and ended up being raise as security incidents.

For SOC 1 their TP rate is 5%, BP rate is 45% and FP rate is 50%.

For SOC 2 their TP rate is 5%, FP rate 95%.

Both SOCs are in the same situation - same workload, 100 alerts, 5 security incidents, but SOC 1 reports a 50% FP rate, while SOC 2 reports a 95% FP rate. Very different numbers for the exact same outcome.

Wrapping up

So that's my list of grievances. Metrics nobody defines the same way, metrics that optimize for the wrong behavior, metrics borrowed from incident response, and resolution categories that make comparing two SOCs impossible. Am I saying we should stop measuring SecOps?

Absolutely not.

In the next article I'll share what those are, based on what worked (and what didn't) in the SOCs I've run or consulted for.

Check out our SecOps Market Landscape tracker and evaluation frameworks

SecOps Market Landscape