It's been almost a year since my post on AI SOC core components. AI SOC is trending more than ever, more teams are evaluating or implementing, so I thought it was time for a refresh.

If you go over the old one you will still that it still holds. Either I evaluated the category well, or the core of this tech didn't move that much. Probably both. What changed is how much clearer the purpose became, and what I expect from each component now.

Let's start with the purpose. Because that's the part that gets lost.

What is AI SOC, actually

Well, you can either read the 30 articles that vendors write, and each of them will tell you why their platform is AI SOC, from their platform perspective, not from a practitioner eye.

Here is the simplest definition I can give, and feel free to quote it:

An AI SOC is a system that combines deterministic automation, machine learning, and large language models to autonomously enrich, investigate, and close security signals. It reaches a verdict on each signal and drives it to an outcome. The level of autonomy can range from AI assisted to fully autonomous.

"Level of autonomy" matters. Full autonomy is the end state. Everything below that is still AI SOC, just with a human closer to the wheel. The industry and the research papers use more or less the same terms, so I will stick to those:

• AI assisted. The human does the work, AI helps with specific tasks. Some papers call this AI in the loop, because the human is still the one making the decisions.

• AI proposes, human approves. Human in the loop. Nothing executes without a human saying yes.

• AI acts, human can veto. Human on the loop. AI runs, a human monitors and can step in.

• AI runs end to end. Human out of the loop. No human in the flow, only in the building and maintenance of systems.

One caveat. Full autonomy is where the vendors are pointing, but it is not where the evidence is. The academic surveys still describe LLMs as augmentation tools and say there is not enough production evidence for autonomous triage yet. So treat the last level as the target, not as something you can buy today.

Pick where you are. Pick where you want to be. Then evaluate against that, not against the vendor's demo.

One more thing on the definition. Many are expecting AI SOC to come with all of this out of the box. I think this is still not clear yet, but that's the expectation from practitioners. I would say if you come with a platform where I need to build this, then that is not AI SOC. That is more an agentic security operations platform that allows you to build solutions. But more on that in a dedicated blog defining the category.

Product Updates !

How to Catch an AI SOC Analyst Bluffing. Register.

~1,200 rogue AI agents breached production infrastructure and spoofed their own logs. Even the investigators' AI couldn't tell the story straight. The same tech is being sold into your SOC as the thing you should trust with verdicts…

Trust none of it.

Sep 16 we show two things: how to catch an AI SOC bluffing, and the first public demo of Morpheus 2. Watch it grade every finding Confirmed, Inferred, or Gap (the bluff lives in the middle category), attach evidence while the alert is still open, and write the playbook without improvising the response.

Detection Eng and AI Threat Intel Ops

Continuous Detection Engineering
Continuously maps detection coverage against MITRE ATT&CK and current threat intelligence, identifies gaps, and generates detections to close them. Finds and tunes the small set of rules driving most alert noise, identifies broken or inactive detections, and optimizes data ingestion to reduce SIEM costs.

AI Threat Intelligence Operations
Analyzes 100+ intelligence sources and correlates emerging threats with your environment to determine what matters, where you're exposed, and whether you can detect it. Relevant intelligence feeds directly into detection engineering, threat hunting, and prioritization.

What it covers

I don't use Tiers here on purpose. I've worked in many SOCs and not two of them had the same responsibilities for Tier 1. So I break this down by responsibility, and you map it to whatever team structure you have.

An AI SOC starts as soon as there is a signal. Then:

1. Enrichment. Context gathering. Get the data you need, what we know about the assets, identities, network, IOCs.

2. Investigation. Answer the who, what, where, when. Build the timeline. Understand the blast radius. Pull more context if needed.

3. Conclusion. Come up with enough evidence to say one of these: benign or false positive, suspicious or malicious, or inconclusive.

4. Closure. Drive it to closure regardless of the verdict.

That last point is where I've changed my expectation. Even when it's malicious, I still expect the AI SOC to drive closure. It should follow your IR process, engage stakeholders, reach out to users, ask the questions, document, take notes, and come up with recommendations. Not full autonomy, but human in the loop or human on the loop.

The only case where it shouldn't drive closure is inconclusive. And if you get a lot of those, your pipeline is broken, not the AI.

What are the outcomes

Each verdict should produce something.

False positive or benign. It should tell me why it thinks so. And it should send me a daily or weekly report: here is what triggered, here is what I closed, here is what you should fix. Detection logic, log sources, whatever it is. If there is nothing to fix, say so.

Suspicious or malicious. Today most people put this in a different tech category (response, IR, SOAR, whatever). I think these will merge. And I think we as an industry should have the expectation that the AI SOC drives some part of IR. Not all of it. But the stakeholder engagement, the user interviews, the documentation, that should not be manual anymore.

Inconclusive. It should tell me what data it needed to reach a conclusion. This is the hardest one to get right with an LLM. Knowing what you don't know is not something these models do well by default.

The core components

Now that we agree on what it does, here is what you need to run it. Same four blocks as last year, updated with what I've learned.

1. Data ingestion

Even though last year I called this data ingestion , it was mostly about alert ingestion. I covered the options I saw across the vendors: platforms that live on top of your detection layer (EDR, XDR, ITDR, email protection, and so on), and platforms that live on top of your SIEM or SOAR and take whatever detection comes out of it. My take then was that the SIEM route usually gives better results, assuming you can afford to get all your data into the SIEM. Still true.

Two points from last year that still hold, and I keep seeing people skip them:

• Not every platform supports every type of detection. The custom or unique rules you built in your environment might not be supported. Check that before you sign, or you create a gap you still handle manually or with your SOAR.

• The AI SOC should try to understand the logic of the rule, not just the alert it produces. Some platforms do this and come back with recommendations. Others just ingest and move on. For me this is not optional.

What changed is the scope. It's not just alert ingestion, it's data ingestion. Alerts, enrichment data, cases, documents, past incidents, anything.

The number one mistake I see: people check that their SIEM, EDR and email protection are supported and think "great, this will work in my environment." That is the easy part. The enrichment and context ingestion is more important than the alert ingestion, and it's where most implementations get stuck.

The exception is if you already have all your context in one place, a knowledge graph or similar. Then you only need to connect that. Not many teams are there.

Side note, and I'm marking it here so I can point back to it later: this is where context lock-in starts. Whoever holds your context graph holds your SOC. Keep that in mind when you sign.

2. Knowledge graph, context graph, whatever we end up calling it

Your SecOps and business intelligence. We should agree on a name for this as an industry, but the name matters less than the thing.

This is the brain. Everything about how well your AI SOC works comes down to this. It needs to hold enough detail and evidence for the investigation engine to reach a conclusion. Asset ownership, identity context, change requests, past incidents, what's normal for this user on this host.

Many vendors say they have it. Not many have it the way it should be. The test is simple: is it machine readable, easy to fetch, and fast? If the AI has to go and query five tools every time to rebuild the same context, you don't have a knowledge graph, you have integrations.

Update- Maxime Lamothe-Brassard pushed back on the "graph" part after I posted this, and he is right. Graph is how we think about it, not how the machine needs it stored. It does not have to be a graph database. Graph DBs are messy and rigid, and a data lake type backend does the job fine and is a lot easier to run. What matters is that the models are aware of the data sources and how they correlate, so the AI can pivot from one source to another based on what it finds. So when I say knowledge graph, read it as the capability, not the storage. I kept the name because that's what the industry calls it, but the implementation is your call. Credit to Maxime for the correction.

I think there will be a dedicated role for this in the future. Someone whose job is tuning the context graph and making sure it works as expected. If that piece works, the rest can be autonomous. If it doesn't, nothing downstream will save you.

3. Investigation engine

Here is what I've seen work: a combination of deterministic workflows, ML, and LLMs (or small models). All three together give the best results.

ML is what gives you speed and accuracy on the boring, high-volume stuff. Deterministic workflows are what give you predictability. LLMs give you flexibility on the ambiguous alerts. You need all three.

What the investigation engine should do:

• Deduplicate and correlate events

• Retrieve the relevant context from the graph

• Ask the right questions

• Pull additional context from external sources if it needs to

• Reach one of the three outcomes as a minimum: benign/FP, suspicious/malicious, inconclusive

• Reassess risk, impact and severity

That last one is not optional anymore. Look at the OpenAI and Hugging Face incident from July. Both sides had the alerts. Both sides failed at a different step.

On the Hugging Face side, their own timeline says the signals from runtime analysis and SIEM were ambiguous on their own. Their AI security stack did correlate them into a coherent attack signal. But it failed to raise the criticality and page the on-call team. Correlation worked, severity didn't.

On the OpenAI side it was the opposite (postmortem, talk). They had an Artifactory outage on July 4 that turned into a security incident. They cleaned it up. Then the agents rebuilt their message board on July 8 and kept going for another eleven days. OpenAI only caught it on July 19 from a privilege escalation alert, and only connected it to the Hugging Face breach the next day when they went to revoke credentials and Hugging Face told them those were already revoked. The signals were there. Nobody, human or machine, put them together.

Correlating is half the job. Re-scoring is the other half. You need both, and in this case each side was missing one.

4. Response and the feedback loop

I covered response above. I see it as part of the AI SOC, not a separate category. At minimum I expect suggestions that make sense for my environment, basic actions (block, isolate, reset, interview the user), and ideally native automation for anything more.

But the piece I want to highlight is the feedback loop. This is one of the hardest parts and not many platforms have it. And again, the reason is the same: it depends on having a good knowledge base. If the AI can't tell what changed between last week's benign verdict and this week's, it can't learn from it.

What I want from the loop:

• Suggestions on how to improve detections

• Suggestions on what log sources or context are missing

• Suggestions on process, where my playbook doesn't match how the investigation actually went

If a platform gives you none of that, it's a faster way to close alerts. And without think you will have broken SOC.

Final thoughts

The components didn't change much in a year. The expectations did.

Ingestion is now data ingestion, not alert ingestion. The knowledge graph is the brain, not an enrichment feature. The investigation engine needs to re-score severity, not just correlate. And response and feedback are part of the package, not a roadmap item.

If you are evaluating this year, start from the purpose: autonomous triage to closure, at whatever autonomy level you're comfortable with. Then check each component against that. The demo will look great either way.

Check out our SecOps Market Landscape tracker and evaluation frameworks

SecOps Market Landscape

This time I’m talking about how »I« think we should approach SOC metrics so they’re meaningful and measure the right things.

BIG DISCLAIMER: you do you. Meaning, define what success looks like and choose whatever metrics can tell you whether you’re on the right track. I’m a big believer that for internal purposes you can measure what you want, as long as it gives you a good idea of where your SOC is and doesn’t demand impossible from the team.

QA goes first

This means that once a week, bi-weekly, or monthly, one of your best analysts samples alerts handled by the team to verify they were done correctly. Correctly here means a couple things:

• validating L1 work more often than not means verifying whether SOPs were followed

• validating L2+ work typically means re-running the investigation

• validating AI work means at least validating the verdict and investigation steps (but potentially much more)

As far as specific approaches, I know SOCs who have found good success in their Quality Assurance efforts by following ISO 2859-1:2026 - it lays out a framework for QA and it’s solid.

Or, here’s what you can do instead:

• QA only high severity alerts (probably a good starting point)

• Sample X alerts per analyst (to spot quality issues with individual team members)

• Sample alerts completely at random

• Sample alerts with the highest MTTR or specific resolution comments

• Query for easily spotted issues like missing resolution comments, wrong tags, or unassigned alerts

Why do we start with QA, you might ask? First of all, as AI is increasingly baked into threat detection tools, chances are your analysts will end up doing more QA than ever before. Good to start them early.

Then, and more importantly, without QA your metrics are useless. What good is MTTA / MTTR in green if alerts aren’t resolved properly or what does the false positive rate matter if alerts are categorized wrong? And so on.

Quality Assurance unlocks the value of your metrics and should always be the starting point. Now - how many SOCs start establishing solid QA fundamentals before worrying about MTT-X? In my experience, not many.

What metrics are good, then?

A boring, yet 100% true answer, is most of them once you've set up Quality Assurance. Suddenly the volume numbers, TP rates, MTT-X mean something.

Ok, we can’t leave it here or it will be anticlimactic. Here are 3 SOC metrics that »I« have been playing with recently and either found good success with or think those are interesting as concepts:

Metric 1: Cost per data source / detection

SIEM TCO is a big number. Often the single biggest item in your security budget or close to it. It’s difficult to discuss without a ton of context, and I found senior leaders to be often either unwilling or unable to grasp it.

And yes, we could do the usual money go up, risk go down. But what if instead we broke the big number down into multiple, individual line items?

And there are so many ways of doing that.

You could first work to understand capabilities of adversaries likely to attack your company based on your individual threat model - can start with your industry, geography, size. List their techniques, see which ones repeat the most, outline top 5-10 and make them your threat detection targets.

Calculate how much achieving each of them costs and track that.

Or, just do a flat $ per data source versus how many detections you have for it. Then maybe add context about whether those have ever fired and if you’re feeling fancy talk about the TP / FP rate.

That’s a lot of words, so let me explain with a scenario:

Your SIEM budget is $20,000 per month. You’re collecting $5,000 worth of firewall logs, you have 5 detection rules that use those logs, and you generated 2 firewall related alerts this month, both false positives. Firewalls are not your critical assets. This means that:

• Firewall logs are 25% of your total SIEM cost

• You’re paying $1,000 per detection rule

• You’re paying $2,500 per alert

Is this cost effective? I’d argue it is not.

Obviously, define what cost effective is for you and track against that. I once assumed that $1000 per detection rule, $500 per alert is the limit, went over all our detections rules and had interesting findings.

Yeah, I know not all rules are meant to fire regularly, but you get me.

And if you can add your own risk context to all of that and say that, for example, to write detections around your crown jewel with an ALE of X$ you need Y$? That’s a very mature way of communicating your budget needs.

Trust me it might work.

Metric 2: Alert volume in relation to quality

In my LinkedIn posts I often ridicule the well established practice of preparing fake SOC workload reports with inflated numbers, to then use them as a marketing prop. I do it so much that if you close your eyes and throw a stone at my content, chances are you’ll hit a post about this very topic.

Alert volume obviously matters only after finetuning and automation. Personally, I like to calculate it per analyst because I know roughly how many alerts an analyst can handle in a day (in-house analyst ~25, MSSP analyst ~50 btw.). But what does it matter if volume goes up one month, and down on another?

Is it good if it goes down? Did we finetune our detections, or lose visibility somewhere? And if it goes up? Did we cover a detection gap, or did a random rule just fire a lot?

The volume itself is not an amazing metric, but what if we measured it in relation to quality? Not only does it make for a more complete metric, I believe this is also one of the better ways to understand your SOCs workload ceiling.

Let me explain.

There’s a breaking point where your analysts will start missing alerts entirely due to high workload. But before that point, there’s a whole range of alert volume that won’t cause analysts to miss alerts, but will impact investigation quality. That’s the danger zone you’re trying to avoid and want to be aware of.

In other words - workload goes up and quality stays intact? That’s fine. Workload keeps going up and quality starts going down? That’s where you probably want to stop.

(Small disclaimer - I mention % of alerts with quality issues as a shorthand to represent a QA number. For some SOCs it will be % of critical defects, defect rate, alerts that passed QA etc. - many ways to present the same idea.)

(Another small disclaimer - obviously quality issues are not there just due to the workload, but, for example, insufficient training. Up to SOC leads to figure that out)

Metric 3: Automation rate

Back when I was still a young Rafal I remember hearing SOC leaders talk about how many alerts their SOAR closes automatically and I thought to myself - “there is no way anyone cares about that”.

Turns out this is a metric that for some reason lands extremely well with senior stakeholders.

No idea why, but here’s what happened to me on multiple occasions in my days as a Security Architect: during a leadership meeting I’d throw out a % figure of how many alerts we’re closing automatically before they reach the SOC. Thought nothing of it. Then a couple weeks after I’d hear from people who were not part of those meetings about how our SOC (or managed SOC) closes 40% of alerts automatically. It was either other leaders or customer success people when I was with an MSSP.

???

Lessons learned I guess. If that figure stays with people, might as well start measuring it. Some ways I’ve done it:

Automation Rate = (Alerts Closed By Automation / Total Alerts) × 100

or

Automation Rate = [((Alerts Closed By Automation / Total Alerts) × 100) + ((Alerts Enriched By Automation / Total Alerts) × 100)] / 2

or

Automation Rate = (Alerts Where SOAR Automatically Kicked In / Total Alerts) x 100

Or any other way. The possibilities are endless, especially with AI increasingly doing triage and assisting in investigations. I’m sure people writing the checks to bring in that technology are interested in knowing how well it works.

And with Quality Assurance established % of alerts handled “autonomously” by AI is a good metric to track.

One important caveat though. Automation is not the same as just auto closing your all DLP alert because someone decided to enable Microsoft Purview and you got 40.000 alerts in your queue the next day. The definition of what constitutes automation is up to you, but I believe at minimum it requires more logic than:

if alert.title == "XYZ":
alert.status = "closed"

What else

Those were the more “interesting” metrics, not the entirety of what I consider worth tracking.

If I were to give a simple summary of what comes to mind here’s how it would look like:

I don’t obsess over (or don’t see much value in tracking):

• MTTA / MTTR for all alerts

• Alert volume before tuning

• Detection coverage (prefer focusing on meeting detection targets instead)

• MTTD (that’s how fast our tools work)

• Escalation rate

• False negative rate (don’t get me started on this one)

• Anything related to how fast detections are written

I like tracking:

• QA (% of critical defects, % of alerts that passed QA, whatever feels right)

• Alert volume per analyst (and in relation to QA)

• Cost related metrics (as described above)

• Automation rate

• TP / BP / FP rate (BP rate specifically as a detection engineering metric, meaning detection good, behavior benign)

• MTT-X for a subset of alerts (mostly high sev)

I’m sure I missed a ton.

Some people say that security metrics don’t matter at all and what matters is that the business can keep achieving its objectives without third party interruptions. Some say the real metric is the friends we’ve made along the way. I believe metrics can be useful, but you need to be selective with what you track.

And most importantly, metrics don’t matter if you don’t have a solid Quality Assurance process in place.

Check out our SecOps Market Landscape tracker and evaluation frameworks

SecOps Market Landscape

Route the wrong stage to the wrong tier and the system fails in a way no accuracy number will show you.

An AI SOC is not one technology. It is at least three, with different cost, latency, and reliability characteristics. The engineering question is not which one to use. It is which one to apply where.

Most products collapse too much into one tier. That is where the failures come from.

The three tiers

Deterministic execution. Code, rules, workflows, lookups. Correct behavior is specified in advance, execution follows explicit logic, marginal compute cost is typically low, and the path is straightforward to audit.

Machine learning. Classification, clustering, anomaly detection, baselining, graph analysis. Operates over a population and finds structure nobody can completely specify in advance. Low cost per item and measurable with precision, recall, false positive rate, and drift.

Language models, large and small. Reasoning over unstructured and contradictory evidence, forming hypotheses, generating queries, producing explanations. Highest cost, highest latency, and the tier whose output is hardest to reproduce exactly.

There is ordinary analytics around these tiers too. A count, aggregation, join, or group-by is not machine learning. If SQL can answer the question exactly, SQL does not need an agentic rebrand.

The architectural principle is simple. Computation with a known operation and a correct answer should happen outside the reasoning model.

Product Updates Section !

Deterministic when it matters

Security teams shouldn’t have to choose between human-speed triage queues and LLM reasoning loops that consume the seconds attackers don’t give back. Wirespeed’s AI SOC prioritizes deterministic execution to handle most investigations and achieve containment in milliseconds. Agentic AI is used where it matters: context and edge cases, not decisions without guardrails. Breakout times are as fast as 27 seconds, and pure-LLM SOCs can burn 15 to 90 seconds per decision before acting. Wirespeed automates Tier 1 to 3 workflows across your existing stack, cutting alert noise without latency, second-guessing the probabilistic results, or exploding token costs that comes from other AI SOC solutions.

Released Delta Tradecraft

Detection logic is cheap now. Detection knowledge is not. A model can write the query. It cannot tell you how the behavior manifests, what evidence it leaves, or whether your telemetry can see it.
Delta Tradecraft is Spectrum's open standard for capturing that research once and making it portable.
Spectrum runs it continuously. New threats researched, customized to your environment, and tested against real logs before deploy. Existing detections retested, root-caused, and repaired as the environment shifts.

Get covered faster. Stay covered after.

Where each one belongs

Deterministic owns repeatable stages and enforcement:

• Parsing, normalization, deduplication by shared key

• Entity extraction against known patterns, enrichment lookups, list checks

• Response execution and ticket updates

• Approval gates, scoping, rate limits, circuit breakers

A guardrail that holds most of the time is not a guardrail. Your account disable action does not need to be creative.

Machine learning owns scale:

• Clustering thousands of related alerts into cases

• Baselining normal behavior for users, hosts, and service accounts

• Scoring priority across a population

• Ranking candidate related cases across long histories

• Detecting drift

Language models own judgment:

• Building a timeline from evidence across five systems

• Assessing blast radius

• Reading phishing bodies, scripts, or ticket comments

• Weighing evidence that points in two directions

• Deciding which tool to call next based on the last result

Small models deserve separate mention. Narrow repeated tasks such as alert classification, observable extraction, and case routing do not need a frontier model. They are cheaper, faster, and easier to evaluate because the task is bounded.

Not every alert needs to visit a data center full of GPUs before somebody decides it is another failed login.

Correlation is where this becomes obvious

The industry uses one word for three very different operations.

Grouping by shared key is deterministic. Same host, same rule, same user, same time window. Most products call this correlation. It is aggregation, and it finds what already shares an identifier.

Statistical and graph correlation is ML. Co-occurrence that exceeds baseline, sequences that repeat across a population, paths through an entity graph connecting signals with no direct key. It finds that two things travel together, not necessarily why.

Attack-pattern correlation is reasoning-led. A failed authentication in one system, a scheduled task created somewhere else, and an outbound connection from a third host may share no direct key and have little useful statistical history. They can still belong to the same case because they fit an adversary behavior chain.

Recognizing that means hypothesizing a relationship, then looking for evidence that confirms or kills it.

Reasoning is not the only way to correlate an attack chain. Rules, graphs, and sequence models can identify parts of one. The difference is that reasoning can hypothesize a relationship nobody encoded in advance and actively investigate it.

Rule-based correlation engines have been shipping since the early 2000s, but somebody has to anticipate the relationship.

We have spent twenty years writing increasingly sophisticated versions of "if A and B happen within fifteen minutes, open a case."

It works. Until the attacker does C.

The number of possible combinations is too large for a team to enumerate. That is one of the constraints reasoning lifts, and it is a better argument for AI SOC than faster triage ever was.

How to decide placement

Four questions settle most cases.

• Is the correct operation knowable in advance? If yes, make it deterministic. Do not reason about it.

• Does the answer emerge from a population rather than an instance? If yes, use analytics or ML. A single alert cannot tell you what is anomalous.

• Does it require judgment over unstructured, incomplete, or contradictory evidence? That is where a language model earns its cost.

• What is the blast radius if it is wrong? Irreversible actions execute deterministically, behind a gate. Reasoning can propose them. It should not be the last thing standing between a proposal and a disabled account.

Frequency does not change the answer.

A judgment call does not become a lookup because it happens a thousand times a day. But high frequency changes the economics.

Decompose the work. Maybe the enrichment in front of the verdict is deterministic. Maybe extraction can move to a small model. Maybe routing happens before reasoning. Move those parts down and keep judgment where it belongs.

This is how you avoid paying frontier-model prices to discover that 8.8.8.8 is Google DNS for the ten-thousandth time.

The tier newer vendors skipped

The incumbents built the ML tier. SIEM, UEBA, and EDR vendors have run clustering, baselining, and statistical scoring in production for years.

A large share of newer AI SOC entrants started at the language model and never built the middle. Not all of them, and some ship real ML without talking about it, because ML had bad marketing and agents currently have excellent marketing.

The math did not care.

A model with a query tool can tell you the same pattern fired 4,000 times this month across 200 hosts. The question was never capability. It is what the answer costs, how long it takes, and whether you can trust it twice.

A group-by over indexed data runs in milliseconds and returns an exact result for the state of that data. Reaching the same result through a model means either asking it to generate the query, which is reasonable, or pulling records into context and asking it to do the computation, which is not.

Let the model write the SQL.

Do not make the model cosplay as the database.

Where the missing ML tier actually hurts is the work that has no simple query to write.

Which behavior is anomalous for this account? What does normal look like for a service principal nobody documented? Which signals travel together across the estate without sharing a key?

Those answers come from computation over the population. If that computation does not exist, the reasoning tier is not filling the gap. It is working without the input.

You can build an AI SOC without a real ML tier. But the work relocates:

• Correlation collapses back to grouping. Hand-written keys and time windows find what already shares an identifier.

• Baselining becomes stored counters. First-seen, frequency, and threshold rules become a thin implementation of statistical baselining.

• Prioritization becomes prompt instructions. Asking a model to rank a queue it only sees one item at a time from is not ranking. It is guessing with good grammar.

• Guardrails multiply. More validation, schema checks, retry logic, and hard limits appear around the model.

This can hold together at moderate volume. At scale, population-level questions either go unanswered or get answered by running a model over work a cheap classifier or statistical system should have handled.

The problem is discovering that architecture at renewal, when the token bill scales with alert volume and nobody can explain how accurate the prioritization is.

Nothing makes architecture visible quite like the renewal spreadsheet.

Why deterministic automation failed before

Every SOAR program has some version of the same history.

A playbook covers the common case. A field changes. An unusual variant arrives. An API returns an unexpected error. The team patches it. Patches accumulate. Maintenance eventually costs more than the work being automated and the playbook gets quietly disabled.

Nobody deletes it, obviously.

It remains in the SOAR forever, named something like Phishing_Response_v2_FINAL_new.

The failure was not determinism. The problem was that every exception had to be anticipated and implemented by a human.

Reasoning changes that in two places.

Runtime fallback. When the deterministic path reaches a condition it does not handle, escalate that case to reasoning. Deterministic execution keeps carrying the volume. You pay for reasoning on the fraction that needs it.

Build-time repair. Coding agents can write and fix the deterministic layer. New parsers, integrations against documented APIs, workflow updates after schema changes. This is bounded work that can be tested.

Together they create a useful loop.

If reasoning handles the same edge case repeatedly, codify it. Move it into the deterministic path and the next execution is fast, repeatable, and cheap.

The system should get cheaper and more predictable as it learns what work no longer requires reasoning.

Agent-written code is still code, though. It needs tests, review, version control, and an owner.

git blame does not accept "the agent wrote it" as an answer.

Each tier fails differently

This is the argument for combining them.

Deterministic logic fails at the boundary of what was anticipated. Good systems fail closed or escalate. Bad ones keep executing because the input passes validation while the assumptions underneath it are wrong.

Machine learning fails on drift. The environment moves since training or calibration and performance degrades gradually rather than breaking visibly.

Language models produce confident, well-written output that can be wrong. Fluency is not calibration.

Combined, they check each other.

Deterministic guardrails constrain what reasoning can act on. ML output becomes evidence rather than a verdict. Model output gets validated against deterministic schemas. Closed case history supports baselining and similarity while giving reasoning grounded precedent.

A system made entirely of rules cannot handle what nobody encoded.

A system made entirely of a model cannot give you the execution guarantees you need.

Putting autonomous in front of either one does not change that.

Reproducibility is a systems property

It is common to hear that language models are inherently non-deterministic. The reality is more interesting.

With greedy decoding, often exposed through temperature-zero settings, the decoder selects the highest-probability next token. Output can still vary. Thinking Machines Lab showed that batch-size dependence in inference kernels is an important source of that variance. Their batch-invariant implementation produced bitwise-identical output across repeated runs, with a throughput cost.

Reproducibility is an engineering choice with a price, not a law of nature.

In an agentic system, the larger source of variance often sits above the model anyway. Tool results change. Retrieved context changes. Multi-step paths diverge. Model and prompt versions change.

By step six, arguing about temperature zero is a bit like arguing about the accuracy of the compass after somebody moved the road.

So do not design around the assumption that a run repeats. Pin versions, validate schemas, test properties rather than exact strings, and keep irreversible actions behind deterministic gates.

None of this works without the data layer

There is another layer underneath all three.

Anton Chuvakin makes this case in his API or Die audit [LINK]. Agents consume APIs at a frequency no human would. An analyst may query a SIEM a handful of times an hour. An agent can do it a hundred times in under a minute.

Run that across an incident and your AI transformation project becomes a load test against your own infrastructure.

Each tier also needs different data.

Deterministic execution needs stable schemas. ML needs representative history, and supervised tasks need reliable labels. Reasoning needs joinable context and structured output.

Hand a model a two megabyte text blob and it spends tokens finding the evidence before it can reason about it.

And access is not retrieval.

MCP standardizes how tools are described and called. It does not make a slow API fast, fix permissions, or repair bad data.

Wrapping a legacy interface in MCP gives you a legacy interface with a manifest.

Congratulations, the technical debt now has a tool description.

Retrieval has the same problem. Security data contains exact facts, relationships, timelines, and prose. Those need different retrieval methods.

A hash does not need semantic meaning. It needs the right row.

Google DeepMind's work on embedding-based retrieval demonstrates fundamental limitations of single-vector retrieval. For a SOC, retrieval needs to be hybrid: exact lookups and queries for facts, ranking for related history, and vector search where the source is actually unstructured.

Retrieval quality becomes a hard ceiling on reasoning quality.

Garbage in, but now with a very articulate incident summary.

So, how do you not kill your AI SOC?

Do not send everything to the biggest model you can afford.

Do not call every group of alerts correlation.

Do not ask a model whether something is anomalous without giving it a baseline.

Do not embed hashes, IDs, timestamps, and every other fact with an exact answer because somebody said RAG.

Do not give the agent 200 tools and hope it develops good taste.

Do not let the same probabilistic system investigate an account, decide it is compromised, and disable it.

And please do not measure all of this with one accuracy number, make the number green, and put it in the QBR.

Use deterministic execution where the operation is known. Use analytics and ML where the answer lives in the population. Pay for reasoning where judgment is actually required. Keep enforcement deterministic.

The goal is not to put AI everywhere in the SOC.

The goal is to know where not to put it.

What to ask a vendor

• Which tier handles which stage?

• If there is no ML tier, what clusters, baselines, and learns from closed cases?

• How does retrieval work? What is queried exactly and what is searched by similarity?

• How is each tier measured? Deterministic execution on coverage and execution success. ML on precision, recall, calibration, and drift. Reasoning on analyst agreement, task success, calibration, and variance across repeated runs.

If everything routes through a frontier model, cost scales with alert volume and peaks when you need the system most.

And if one accuracy number describes the whole architecture, ask what exactly it measured.

Explore the vendor landscape The full map of the AI SOC market, updated as it moves.

AI for SecOps Landscape

Check out our SecOps Market Landscape tracker and evaluation frameworks

SecOps Market Landscape

We have a questionnaire - 20 questions, 5-10 minutes to finish. All anonymous. Asking about SOC workload, team size and composition, automation, daily operations and similar. We publish the full thing for free together with questionnaire so that you can judge the wording for yourself.

If you’re A SOC lead (or were one until recently) we’d love if you took part. We’re not publishing the link openly to keep results clean, so if you’re interested please reach out to us at contact@secops-unpacked.ai or via LinkedIn:

• Filip: [LinkedIn]

• Rafal: [LinkedIn]

• Ignacio: [LinkedIn]

Have you noticed how SOCs are presented as inherently dysfunctional by cybersecurity vendors? If you’ve been to LinkedIn, I’m sure you’ve seen some version of “SOC analysts are facing 10 000 alerts per day”, “65% of analysts confess to ignoring an alert that later turned out to be a security incident”, “Over 50% of all alerts SOCs are facing turn out to be False Positives”, “71% of SOC analysts report burnout“ etc.

And there is always a report to support those numbers. Today I write specifically about those reports and explaining why I think we should not pay too much attention to them.

I’m also venting a little.

Not very zero-trust of us

In an industry built on skepticism, it's surprising how easy we trust in reports.

A prime example is the widespread coverage of the ISC2 Cybersecurity Workforce Report. It seems to be taken as gospel without consideration that one of the largest certification bodies in cybersecurity might have a vested interest in the public believing there’s a cybersecurity skill shortage.

SOC workload reports are no different. If anything, they’re worse. I’m convinced that most SOC related reports (all I’ve seen, can’t claim to have seen them all) are misleading readers on purpose.

Here’s exactly what is wrong with them

The numbers. What do they mean?

Probably the most used SOC workload number is the alert volume. No wonder, it speaks to most people better than MTT-X, TP / FP rate or detection coverage. It’s an easy message to digest and an easy, scary number to wave around.

What are those numbers?

Links from the table: Anvilogic blog · Cisco report · Crogl’s website · Cyberreason’s blog · PaloAltos blog · Thehackernews coverage of Prophet’s report · Trellix’s report · Vectra’s website · Vectra’s report · Fortinet’s report

Those are just 10 random reportings on SOC alert numbers. As you can see, the difference between those numbers can be massive. On one side we have report by Anvilogic with a reasonable looking number of ~286 alerts / day. On another, there’s Fortinet with a claim of 150,000 alerts / day, which is insane.

Just working with this list we could push any narrative we chose.

• Want to show that SOCs are doing ok? Take Anvilogic’s number.

• Want to sell your AI SOC product to a reasonable buyer? Pick one of Vectra’s numbers, both look decently plausible to an untrained eye.

• Want to embarass yourself and show that you don’t know the difference between an event and an alert? Go with Fortinet’s numbers.

You’re probably wondering how we can have a ~525x spread for the question of - what is an average daily SOC alert volume. I don’t know, but I have a hunch. Hear me out.

Duplicates

Most SIEM data tables are immutable - meaning you can’t change records that are already there, you can only add more. In practice, every change to an alert - tagging, commenting, assignment, change in status etc. creates a new record in the table that holds all security alerts.

Now, if we don’t account for those duplicate values and simply query for all records in that table, we will end up with 3-4x the amount of alerts than we actually have.

Pre-finetuning numbers

If you enable your DLP alerting today, tomorrow you'll see 10,000 new DLP alerts in your SIEM. It is what it is. Threat detection tools can generate an absolute mountain of alerts, and the job of SOC teams is to tune that number down to a manageable level.

Now, that means your SOC can receive 10,000 alerts every day, but it doesn't mean your analysts need to go through all of them. Likely 90% of that volume is tuned out, autoclosed, aggregated, you name it.

Which means the question "how many alerts do you receive every day" should really be "how many alerts do your analysts actually have to investigate every day." I can't prove it, but I have a strong suspicion that many of those inflated numbers come from the fact that some SOCs share values prior to tuning.

In-house and MSSP numbers mixed together

In-house SOC analysts and MSSP analysts work differently. In-house analysts typically handle an alert end-to-end (or at least a larger portion of it) and have other tasks as well, like monitoring for user-submitted tickets. MSSP SOCs often go through a significantly higher alert volume in a day. That's because instead of fully investigating alerts, they're often expected to escalate them to clients for further investigation, and they typically don't have other tasks outside security monitoring.

In-house SOCs work for themselves; MSSP teams are typically larger and work for multiple clients. Now, based on what I know about SOC work, I'd say an in-house SOC analyst can handle about 20-30 alerts per day. An MSSP analyst probably handles close to double that.

So the daily alert volume of a 100-analyst MSSP SOC with 20 clients can be 5,000. At the same time, an in-house SOC with 4 analysts might see closer to 100 a day. Both numbers are realistic, but if I mix them together and say that a SOC on average handles 2,550 alerts every day, we're being misleading.

Team size

Similar to in-house vs MSSP - bigger teams are typically dealing with more alerts. We can’t compare the overall daily volume between a team of 5 and a team of 30, regardless of in-house or MSSP. This is a no brainer, but also rarely accounted for.

Straight fraud

All those problems exist before we even consider the possibility that those SOC alert volumes are fabricated. Because, why not? Nearly every one of those reports is paid for and then used as a marketing prop. It's not a stretch to assume the numbers are tampered with in at least some cases. The incentive is there, and we never see the raw data behind any of those surveys, just the conclusions.

So what do we do

Not much we can do, honestly.

The reality is - as long as there is incentive to present inflated numbers, inflated numbers will be presented. Which is why the only way to publish a reliable report I can think of is having practitioners do it.

This is what we will do, and where YOU can help. This only works with enough SOC leads behind it, so if you’re one (or know one) do reach out!

Check out our SecOps Market Landscape tracker and evaluation frameworks

SecOps Market Landscape

Safe to say I’m responsible for a metric (hah) ton of SLA reports and yet, somehow it always felt like those gave no valuable insights into how our SOCs actually operated.

In this blog post I’m highlighting issues I have with how we measure the performance of SecOps, in a follow up article I’ll share my ideas on how we can do it better.

This is part of my healing process so bear with me through this lengthy rant.

Everybody understands metrics differently

I encourage you to do a little field research. Go ask 10 people what MTTR is and how they measure it. Chances are you'll get 10 different answers.

For some the “R” in MTTR stands for resolve. That mostly means a full end-to-end handling of the alert. Sure, that’s actually how I understand it as well.

For others it’s MTT-Respond. But how do we understand response? For some it’s the initial acknowledgement of the alert. That’s also often measured as MTTA or Mean Time To Acknowledge. For others, response ends at an alert being escalated to a higher tier or at an initial notification to the client.

Then, we have a group who claims that “R” in MTTR stands for anything from “Repair”, “Restore” to “Remediate” or “Report”. All answers I got from SOC leads I talked to in the recent months.

Also, it’s no wonder that different interpretations of MTTR exist because MTTR is genuinely different for MSSPs and in-house SOCs. While in-house SOCs mostly measure it as the full time it took to close an alert, MSSPs are often responsible for the initial part of the investigation (triage + maybe some L2 work) and will measure MTTR only for their portion of alert handling.

This means that MSSPs will often see shorter MTTR and higher alert volumes than in-house SOCs even though they might have a common understanding of what MTTR is.

This is just one example, but it demonstrates how a single metric can have multiple different meanings, depending on who you ask. This is why when I hear “our MTTR is 30 minutes” this tells me absolutely nothing.

Overemphasizing speed and volume

Sometimes it’s not even if we understand those metrics the same way, but if those metrics are genuinely useful for us. After all, what you measure is what you optimize for.

And what do SOCs often measure? Speed and volume.

So what do those SOCs optimize for? Speed and volume.

Doesn’t this encourage surface level investigations?

On one hand, we say we want quality investigations. On the other hand, we judge analysts by how fast they handle alerts or how many they can fit into their shift.

You see how that makes no sense?

Some would argue that it’s smart for MSSPs to emphasize speed over quality if they handle the initial triage, and sure, maybe that’s the case. In my experience that can turn into becoming a mass escalation engine, and I know that clients absolutely hate that.

And look, I get it. For a non-technical stakeholder who is used to the “traditional” SOC metrics, having a short MTT-whatever is probably of paramount importance. But I believe this is a slippery slope. Short SLA commitments are often what “sells” your service to a client, but also makes your analysts’ lives that much harder. And if you keep adding clients and all of them need their alerts handled within a very short window, you then have an army of analysts trained to look for a first sign an alert is a false positive. In other words, you groomed your SOC into being mediocre.

Oh, and don’t get me started on alert volume. Those mean nothing in isolation. As in, what is a good number of alerts?

• If the number is high, does it mean that we did a good job investigating, or that we’ve done a terrible job at finetuning?

• And what if it’s low? Does that mean that we’ve finetuned our environment perfectly or that we have visibility gaps?

Alert volume is probably the most frequently cited number when trying to demonstrate that SOCs are doing poorly (FUD marketing doing well as ever). It means that you see claims like “SOCs are facing 10,000 alerts / day” in every other post from a cyber vendor and that is often parroted by the less experienced practitioners.

What’s genuinely upsetting is that each of those outlandish volume claims can be supported by an official sounding report. Pick a number of alerts you think a SOC is handling each day, then look for a report justifying it. Chances are, you will find it:

Links from the table: Anvilogic blog · Cisco report · Crogl’s website · Cyberreason’s blog · PaloAltos blog · Thehackernews coverage of Prophet’s report · Trellix’s report · Vectra’s website · Vectra’s report · Fortinet’s report

There are many issues with the methodology of those reports, probably the biggest one is that when talking about alert volume you absolutely need to discuss team size. 400 alerts per day for 2-person SOC and 20-person SOC is entirely different workload. Add to it how some of those numbers are taken pre-finetuning, mixing up MSSP and in-house SOC numbers and a real chance that some of those numbers are made up and you’ll see why I’m super skeptical when a vendor says “SOCs are facing 10,000 alerts / day”. Sure.

Incident Response metrics mixed with SOC metrics

One thing I’m noticing recently is that many SOC leaders mix up Incident Response metrics with regular SOC metrics. Just this week I’ve seen two comments about MTT-Contain in the context of Security Operations. H…how much are you containing to derive a mean time from it?

What’s even more curious is MTT-Detect. To me this is an incident response metric that describes the time from the initial compromise to its detection. Otherwise what are we measuring? How fast our tools work?

Think about it, we don’t detect anything ourselves. We configure the tools that do so. Our ability to influence the time it takes our tools to detect is limited. We can maybe make detection rules run in a shorter interval, optimize data pipelines, account for ingestion latency, sure. But calculating the time from the original event to the alert firing for every alert (mean) and using it to demonstrate your SOCs success? I’m not buying it.

Oh, and I’ve recently seen SOC leads talk about MTT-Eradicate so there’s that. Again, just how much are you eradicating, good sir?

The curious case of Benign Positives

“True Positive. False Positive. True Negative. False Negative. Long ago, the four metrics lived together in harmony. Then everything changed when the Benign Positives attacked.”

If you’ve worked in a SOC (I assume you have since you’re reading this) you probably saw some type of this sad table that explained how alerts are categorized:

Cool. Actual attack? True Positive. Alert fired for some random activity? False Positive. See how easy that is?

Or rather was, because vendors started adding their own alert resolutions, like for example Benign Positive. And sure, I understand why. Sometimes an alert fires for an activity that looks malicious, but upon investigation ends up being benign.

On one hand we probably needed a way to categorize alerts that require investigation but aren’t malicious to separate them from clear False Positives where you can just close them after a quick look around. On another, that ends up being confusing.

Why you ask?

Consider two SOCs. SOC 1 has a SIEM that uses Benign Positive alert category. SOC 2 operates only on True Positive and False Positive statuses.

Both SOCs received 100 alerts, 50 were clear False Positives, 50 looked malicious and required further investigation, but in the end only 5 were malicious and ended up being raise as security incidents.

For SOC 1 their TP rate is 5%, BP rate is 45% and FP rate is 50%.

For SOC 2 their TP rate is 5%, FP rate 95%.

Both SOCs are in the same situation - same workload, 100 alerts, 5 security incidents, but SOC 1 reports a 50% FP rate, while SOC 2 reports a 95% FP rate. Very different numbers for the exact same outcome.

Wrapping up

So that's my list of grievances. Metrics nobody defines the same way, metrics that optimize for the wrong behavior, metrics borrowed from incident response, and resolution categories that make comparing two SOCs impossible. Am I saying we should stop measuring SecOps?

Absolutely not.

In the next article I'll share what those are, based on what worked (and what didn't) in the SOCs I've run or consulted for.

Check out our SecOps Market Landscape tracker and evaluation frameworks

SecOps Market Landscape

AI SOC, or Agentic SOC as some call it now, went through an interesting evolution in the last 3 years. I has a chance to implement it in an enterprise in the early days when no one trusted it will be a thing, I advised most of the AI SOC vendors as a consultant, I tracked and analyzed almost every vendor in the space as an analyst, and now I build and market in it. So I've seen it from most of the angles that matter.

Enough bragging. The point is the space changed in a very short period of time. What started as a product category is now commoditized and used across the majority of SecOps tooling in some shape or form.

And that's not a bad thing. But let's first understand why it happened.

Product Updates Section !

Agentic Investigations from Triage to Forensics

Spacewalk handles the full investigation depth, from pre-L1 triage to L3+ incident work. It runs agentic investigations inside your SIEM and EDR, and when the case needs it, goes down to forensics: disk images, event logs, even PDFs. What caught my attention is how it handles complexity. The deeper the investigation, the more rigor it applies. It uses Analysis of Competing Hypotheses: benign versus malicious hypotheses, blind-evaluated, and the evidence has to actually discriminate between them. Not just a confidence score slapped on a verdict. The end result: verdicts you can audit, at whatever depth the investigation went.

New Feature: AI Security for Claude Enterprise

AI tools are becoming part of the enterprise attack surface, but most SOCs still have limited visibility into what users, agents, and connected tools are doing. Daylight now provides managed AI security for Claude Enterprise as part of its MDR service. They build and continuously refine detection coverage for AI-native threats across Claude activity, including malicious or unauthorized MCPs, risky Skills and Plugins, prompt injection attempts, and more. Daylight’s agentic MDR investigates each signal, using context from identity, endpoint, cloud, SaaS, and business systems. Daylight other cool product features are the agentic threat hunting and a security data lake.

The SecOps Shift Map

To track the space, I created the SecOps Shift Map. It explains the areas where SecOps tooling is developing. Soon is making 1 year since I first published it and now we have version 2.

Here's how is structured.. On the far left you have the data pipeline and log ingestion. To get alerts, you first need visibility. Then you have the detection layer, plus what I now call SecOps resilience: monitoring your SecOps pipeline itself. You got the alerts, now you need to get alerted when something is malicious or suspicious.

Then you get into triage. This is where you do the analysis and decide how to handle an alert. If it's a false positive or benign, figure out what needs fixing. If it's a true positive, you continue to remediation.

Quick note on terminology: I try to use "response" less these days. For some people, response means you started analyzing an alert. For others, it means remediation, mitigation, containment. Too confusing.

(The map is still work in progress. I haven't figured out where to place attack simulation/emulation or deception tech yet. My current thinking is attack simulation sits under SecOps resilience, because you're testing your pipeline.)

Here's the kicker

Most AI SOC vendors started in the middle of the map. Triage. And to some extent, that's how the product category got shaped.

In my mind, that was never going to be enough. Something that just does triage will get commoditized. And it did. Most SIEM, EDR, SOAR, XDR, or whatever tech SecOps teams use, added some form of autonomous triage. Some of it is simple, mainly throwing stuff at an LLM. Some of it is complex agentic systems with RAG and automations.

I won't argue about what "true AI SOC" is, because we're not there yet. When I speak with practitioners, no two give me the same definition. It means different things to different people. So I stick with AI SOC as we know it: AI/LLM and agents used for autonomous alert investigation. Based on that definition, it got commoditized.

The prediction that played out

What I said last year: because of this, it won't be enough for AI SOC vendors to sell just triage. They will move either left or right on the Shift Map.

Right means the SOAR route. Add response, some level of automation builders. Left means threat hunting, detection engineering, or even building a SIEM.

And exactly that happened. You can see how the Shift Map looked last year versus how it looks now.

Ah, and one more shift I almost forgot. Some vendors will go into service offerings and add MDR. We have that as a separate highlight on the Shift Map, since that one is services rather than tech.

The analysts are catching up

This connects to something I covered in a recent Friday SIEM post: the movement isn't only coming from the AI SOC side. Data pipeline vendors are moving into SIEM and adding AI SOC on top. Cribl acquiring CardinalOps is the latest example. The AI SOC piece is what makes them compelling. In the past you needed SOAR-like capabilities to offer the investigation piece, now the agents cover that.

And the demand explains why everyone converges. Many orgs looking into AI SOC don't just want triage. They want to replace or consolidate their SIEM, MDR and SOAR costs. Every AI SOC conversation is a consolidation conversation.

The analyst firms see it too. Gartner even created a new category for it, called ISOC (Integrated Security Operations Center) Solutions. Forrester took a different route. In their latest XDR Wave they kept the XDR name but added AI agents and agentic systems as a separate evaluation criteria, and SIEM replacement went from experimental to reality. So basically, XDR with agents.

In my view, this is where many of today's players around SecOps will evolve.

Will it change the name? Not so sure. Gartner invented a new one, Forrester stuck with XDR. And even XDR failed to replace SIEM as a term.

How I see this playing out

Three types of implementations:

• Platform play. This is where the ISOC category is getting stronger. The usual large players that offer everything.

• Point solutions and decoupled SIEM. Many smaller solutions added to the stack, a combo of vendors plus build it yourself.

• Existing stack plus MDR and/or AI SOC on top. For those that want minimal tech disruption. And here I think the demand is high: AI SOC with MDR, or MDR with AI SOC capabilities.

So, pure play or add-on?

You might be wondering what the advantages are of pure play vendors versus the ones offering AI SOC as an add-on capability.

Well, it depends what you're looking for. As in many markets, you'll always have niche pure play players versus the big ones that have a bit of everything. Some capabilities are there just to tick a box, others are full blown.

For pure play, I think the advantage is focus. They concentrate on that one thing, so they put a lot of effort into developing the autonomous investigation. For the vendors adding it as a capability, it might be a checkbox, or it might be a full blown capability. You have to check.

I won't share my view on which is better. Some might say I have bias here, so I won't say one beats the other. That's the whole point of SecOps Unpacked: we give you all the tools to evaluate, and you tell us which one you find better and why.

The plot twist I didn't predict

Vendors that were not AI SOC rebranded as AI SOC. And on the other side, AI SOC vendors rebranded as something else, like SIEM.

We track that too. One insight into how we map vendors: if a vendor was founded before Gen AI, so before 2022, they get classified as AI SOC as a capability add-on. That way you know the underlying tech is something else and there was a shift in technology. I think this is fair for everyone.

On top of all this, there are teams building AI SOC in house. Too early for success stories there.

Do we have too many vendors doing AI SOC?

Maybe. But compared to vendors doing security for AI, it's about 3 times less. So I wouldn't call it crowded just yet.

Where to go from here

On the SecOps Unpacked vendor tracker we have close to 140 vendors, all mapped to the Shift Map, so we can track the changes live year over year. Go check it out and let me know what you think.

The full mapping of all vendors per category will be released in our AI for SecOps Market Research coming in November.

Check out our SecOps Market Landscape tracker and evaluation frameworks

SecOps Market Landscape

TL;DR. Every AI SOC vendor ships a "10 questions to ask" checklist tuned to their own product. We built a vendor-neutral one instead. ASEF, the AI SOC Evaluation Framework, scores any AI SOC platform across the whole lifecycle: data ingestion, detection, investigation, response, plus a trust layer underneath.
Score all of it for a fullness reading, or scope to the one zone you are buying and score just that, in depth, with no penalty for the rest.

No product behind it, no pricing, your scores stay yours. It is live and free at secops-unpacked.ai/asef/guide.

AI SOC / Agentic SOC is everywhere - The phrase currently covers everything from fully autonomous investigation pipeline to a chatbot that suggests you maybe reset a password. Same label, same analyst category, same booth. And when a security team sits down to compare three of these products, there is no shared way to measure the distance between them.

Back in February we published ARMM (AI Response Maturity Model), a maturity model for AI SOC response capabilities. It did its job, and it stopped at response on purpose. The limitations section said it plainly: detection and analysis are out of scope.

The thing is not many buy just response product. They buy an AI SOC platform, and that platform it should touch everything (here we might be going into the ISOC term). The data pipeline, the detections, the investigation, the verdict, the action, and the loop that feeds lessons back into better detections.

So we extended it. ASEF, the AI SOC Evaluation Framework, is a vendor-neutral framework for evaluating AI SOC platforms across the whole security operations lifecycle: data ingestion, detection, investigation, and response, plus a cross-cutting Platform and Trust layer. That is the one-line version. Let me walk you through the rest.

Vendor Updates Section !

Agentic AI SOC Platform

Built by SecOps, for SecOps, Prophet Security's Agentic AI SOC Platform elevates every part of your SOC. AI SOC Analyst dynamically builds an investigation plan; correlates evidence across identity, endpoint, cloud, email, and network at investigation time; and reaches a defensible determination in minutes, with the full reasoning trail behind every call.

Across customers, they are seeing 90%+ reduction in noise and false positives, ~4 minutes MTTI, and thousands of analyst hours saved, enabling analysts to improve detections, proactively threat hunt, and focus on improving their organizations' security program.

Agentic Operations Layer

Every AI SOC tool promises autonomy. Few run on a foundation that provides verifiable results. Strike48's Agentic Operations Layer closes that gap with hybrid deterministic and cognitive workflows that make every action repeatable. Introduce agents at your own pace, with no rip and replace of the tools you already run. Configurable human-in-the-loop controls keep you in command. Full audit trails give you visibility into every decision and action. Underneath it all, real-time data federation gives you a complete view of logs across every system, providing the foundation agents need to take action you can trust.

The market rushed to the middle

Here is what actually happened over the last two years. Everyone built the middle. Triage and investigation were the easier win, so that is where the products piled up. The demos are impressive. Alerts get worked fast. And both ends stay thin. Very few products can improve your detections. Very few can execute a remediation without three humans watching. At SecOps Unpacked we track this market closely, and you can see the clustering across the whole vendor landscape. The market optimized for the part that demos well.

We are not the only ones seeing it. Anton Chuvakin and Oliver Rochford earlier this year published When Marketing Fails, a paper on the gap between AI SOC vendor claims and what practitioners actually experience. Their finding, from 30 plus vendor briefings: most claims describe a future state sold in the present tense, and when reality disagrees, the product immaturity gets reframed as a buyer readiness problem. Their fix is to separate assistive from autonomous capabilities, measure adoption depth instead of feature enablement, and treat trust as something a system earns.

ASEF has a mechanism for each one. The autonomy scale separates assistive from autonomous, per capability. Automation depth measures depth, not enablement. And the trust metrics live in the ROI panel, not on the slide.

What ASEF actually is

ASEF is a funnel. Five stages, and each one does exactly one job.

Screen. Narrow the market to a shortlist using your scope and your hard requirements. Requirements are binary. They filter, they never get scored. The idea is that if you want platform that does only Detection Engineering, you don’t need to evaluate the one that does only data pipeline.

Score. Score the survivors capability by capability, per zone, on an autonomy scale.

Platform. Score the cross-cutting Platform and Trust layer separately. It never blends into the headline.

ROI. Track operational metrics as deltas against your own baseline.

Decide. Read it all together against thresholds you set before the demo.

The stages stay separate on purpose. Most bad evaluations happen when these jobs collapse into each other. A requirement becomes a score. A platform gap gets averaged away by a strong feature. A single blended number hides the weak zone. ASEF refuses all three.

Four zones, left to right

Everything hangs on the Shift Map.

Data ingestion and processing. Can it onboard sources, parse, normalize, and tell you where your data gaps are.

Detection engineering and SecOps resilience. Can it author detections, map coverage to ATT&CK, run a proactive hunt off a hypothesis and turn what it finds into a detection, tune the noise, and manage detection as code.

Investigation and Triage. Can it build context, enrich, correlate, scope, and land on a verdict you can defend. Can it run a reactive hunt off an indicator, sweeping the environment for the same activity. And when the verdict is real, can it go deeper. Memory analysis, artifacts, root cause, evidence handling. Triage depth, hunt depth, and DFIR depth are not the same thing, and this zone scores all three.

Response, Remediation, and Feedback loop. Can it act, how autonomously, and does what it learned flow back into better detections. This zone is ARMM, all five planes, plus the feedback loop.

Under all of it sits Platform and Trust. Audit trail, reasoning logs, RBAC, governance, model handling. It gets its own scale and its own card, and a platform score below 50 percent raises a risk flag that no feature strength can clear.

There are 126 capabilities in the seed set, and the whole thing is data-driven. A new capability is one data entry, not a code change.

Question one: what can it do, and how autonomously

Every lifecycle capability gets scored on the autonomy scale we carried over from ARMM.

0 means no capability. 1C means the AI collaborates and the analyst does the work. 1G means the AI lays out options and the analyst picks. 1A means the AI prepared the action and waits for a human to approve it. 2 means it runs end to end, no human in the loop.

Level 2 is still rare, and that is fine. The scale exists to show the distance between the marketing and the autonomy, and to show which products are actually moving.

Two readouts fall out of this, and you read them together. Coverage, the share of capabilities above zero. And automation depth, the distribution across the levels. High coverage with low automation is a guided workflow tool with AI branding. Moderate coverage with real autonomy where it counts is a different product for a different buyer.

Question two: how hard is it for your team, and what breaks if it is wrong

Builder mode scores the same capability for your reality, across three axes. Trust, how much confidence the implementation deserves. Complexity, how hard it is for your team to build and run. Impact, the blast radius if it goes wrong. Add them up, 3 to 9, and the score maps to a tier.

The same capability lands at a different tier for a mature team than for a junior one. The capability is identical. The context is not. A vendor benchmark alone was never going to capture that, and that is exactly why Builder mode exists.

A profile, not a single number

ASEF does not produce one blended score, because one blended score is exactly how a Middle-only product gets to call itself mature.

The headline is a profile across the zones you chose to evaluate. The composite label is gated. A product earns a full label only when every in-scope zone clears the tier and a feedback loop exists. A product that is Expert at investigation and empty on both ends gets labeled Middle-heavy. On purpose. That label is the whole point of this framework turned into a score.

Scope matters too. You pick which zones you are evaluating at the start. A team that handles detection elsewhere and only needs investigation and response does not get marked down for skipping the left side. The report opens by stating the scope, and out-of-scope zones read as not evaluated, never as gaps.

And scope changes the reading, not just what shows. Evaluate all four zones and you get the fullness score, one composite across the lifecycle. Evaluate one or two and you get each zone scored on its own tier, in depth, down to its subdomains. No composite, no blend, no penalty for what you did not ask about. If you are buying an investigation tool, you get an investigation score, not a lifecycle grade that docks it for lacking response. Zones never blend, because blending two of them into one number recreates the same hiding problem one level down. The Platform and Trust layer is scored either way, because it applies to any platform you buy.

Unknown is not a pass

The screening stage has one rule I care about more than any other. A vendor fact is either present, or it is unknown. Unknown never quietly becomes a pass or a fail.

Every vendor in the screen lands in one of three states. Passes, on known facts. Excluded, because a known fact fails a hard requirement, hidden behind a deliberate reveal and labeled with the reason. Or unknown, meaning a required fact is missing, shown by default and flagged as verify in a proof of concept.

Most comparison spreadsheets quietly treat missing information as either fine or disqualifying, depending on who built the spreadsheet. Both are wrong. Excluded fails a known fact. Unknown might pass once you verify it. Keeping those two apart is the integrity of the whole gate.

And the same honesty applies to the AI itself. Chuvakin and Rochford close their paper with the question every buyer should push a vendor on: can the system admit it does not know, and what happens when it does. ASEF scores that directly. Inconclusive verdict support is a capability in the Investigation zone, because a system that forces a binary call on thin evidence is not confident, it is reckless. Override telemetry and published failure modes are capabilities in Platform and Trust. A vendor that treats analyst overrides as user error and documents no limitations does not have a feedback loop. It has a narrative.

Is the SOC getting smarter, or just faster

The last layer is PICERL, 15 metrics across the six phases of the SANS incident response lifecycle: Preparation, Identification, Containment, Eradication, Recovery, and Learning. The phases are SANS's. The index built on them is ours.

You record a baseline before the proof of concept and track deltas against it. No baseline, no evaluation. And there is deliberately no single ROI number. It is a panel of deltas, so strong movement on one phase cannot hide a regression on another.

The metrics that matter most are not the speed ones. Auto-close reversal rate. Escalation accuracy. Model drift. Whether analyst corrections actually feed back into the system. Closing alerts faster is sweeping the floor faster. The question that matters is whether the SOC is getting smarter.

What ASEF is not

No vendor scores. The directory holds facts, and any editorial reference is clearly marked context, never the verdict. Your scores are your own.

No pricing. Because that is changing too often, it is up to you to take this within your org and compare what you get for the price that you pay for the product.

No integration coverage module yet. Integrations are breadth, not autonomy, and they deserve their own percentage-based score. It is on the roadmap. Until then, integration quality gets tested where it should be, in the proof of concept.

No environment tailoring yet. A future version will let you pick your log sources and your top use-cases, scope the capability set to your reality, and surface your telemetry gaps. The design exists. The module does not.

Bottom line

The framework reference and the interactive guide are live at secops-unpacked.ai/asef/guide. Screen the market, score a product, export the results. ARMM stays a standalone model at armm.secops-unpacked.ai and now also lives inside ASEF as the Response zone.

Thanks to Andrei Cotaie and Cristian Miron. Their work on ARMM is the foundation this whole thing stands on.

ASEF is data-driven by design, so contributing is a small edit. A capability, a metric, a vendor fact correction. Send it through the repository or the site contact form. Every contribution says what to measure, never how good any one vendor is at it.

No current product will score well across the whole map, and that is not the point. The point is a common language for what "AI SOC" actually means, so the next conversation you have with a vendor is grounded in capabilities and autonomy instead of promises. The framework will move as the market does.

Check out our SecOps Market Landscape tracker and evaluation frameworks

SecOps Market Landscape

Over my career I’ve participated in and / or led ~30 SIEM onboarding projects, probably closer to 50 if we count individual deployments (1 client, multiple SIEMs type of deal).

Eventually I worked my way towards a SIEM onboarding approach which I will be sharing in this article. It is data driven, environment agnostic and works well for both MSSPs and in-house SOCs.

Let’s go.

Why have an approach

Why can’t you just common sense your way to a solid SIEM deployment and, by extension, a decent security monitoring strategy? After all, you’ve (hopefully) worked with a SIEM before, you’ve seen the detections, queried the data. Why shouldn’t you just sit down with your team and try to just… figure it all out?

Good question.

For MSSPs that’s obviously a bad idea because you’d be turning each engagement into a bespoke one and you don’t want that.

For in-house SOCs it might work, but I don’t think it would be a long term solution. Ideally you’d want your security monitoring approach to outlast the current team.

I doubt you can common sense your way to a solid security monitoring strategy. Mostly because, we’re all biased whether we admit it or not.

What I saw working on my 30+ SIEM onboarding projects is that without a solid, data driven approach, the project plan defaults to the perspective of the most senior stakeholder. Roughly this:

STORY TIME START

I once worked on a SIEM onboarding project for two almost identical companies. Same industry, same size, same technology stack. Both ended up with completely different SIEMs. This was purely because the CISO of the first company used to be a sysadmin, while the SOC manager at the other had a purely networking background.

One SIEM ended up being heavy on endpoint data and detections; the other prioritized network visibility. To this day, I'm not sure which one was more effective at detecting the right threats.

STORY TIME END

There are other reasons too, but I don't want to go into depth or this article turns into a book. For example:

• Staying cost-efficient: without a clear security monitoring strategy, you tend to over-collect data and make your SIEM bulkier than it needs to be.

• Keeping alert volume low: having a clear priority on what needs to be alerted on allows you to be surgical with fine-tuning.

• Being up to date with the world: making sure your ability to detect threats matches the modern threat landscape.

My thesis is that you can’t common sense your way to a solid security monitoring strategy. I believe you need an approach that’s data driven, reusable across different environment and ideally one that can be easily updated to account for new attack vectors.

Here it is:

Chunking the project down

SIEM onboarding or migration is a large project. And what do we do with large projects? We chunk them down into manageable pieces.

Here’s how I do it.

I chunk the SIEM onboarding project into three phases, each taking roughly 30 days. Once the project is complete, you (or your client) ends up with a fully functional threat detection system.

Obviously, threat detection and response content needs to be updated on a reasonably frequent cadence, but that is a story for another time. Let’s dig in:

Pre-onboarding

Two of the most common questions I’d get when onboarding clients to a SIEM was:

• What should we monitor?

• What data do we need to collect?

Some would argue (I’m some) that answering those two questions solves 80% of your security monitoring strategy. The approach that worked the best for me, and one I’d like to instill in you, is this:

Threat intel informs you what detections you need. Knowing what detections you need, it's easy to decide what data to collect (at least the hot-storage portion of your data).

Let me repeat:

1. Threat Intel

2. Detections

3. Data

In this order. Here’s how to operationalize it.

First, you want to understand the monitored environment. If you're an MSSP, you can send a questionnaire to your clients. Figure out the number and types of systems, the existing security stack, maybe also regulatory commitments.

Then you want to prepare a threat landscape report. Nothing complicated:

1. List threat actors who are most likely to attack you or your client. This is based on factors like industry, geography, size, history of past incidents, and questionnaire input.

2. List MITRE techniques most used across those threat actors.

3. Map those techniques in a MITRE Navigator or similar tool for a clean presentation:

Your goal is to list the top 5–10 MITRE techniques most likely to be used against you or your client. The next step is drafting detections for those techniques and understanding what data needs to be collected to support running those detections and the subsequent investigations.

Once this is done, you’re ready to move to the first phase of SIEM onboarding: centralize & learn.

[Phase 1] Centralize & learn (30 days)

The first phase of SIEM onboarding is straightforward. You grab alerts generated by your other threat detection tools and send all of them to your SIEM. And yes, I expect you to have a bunch of tools deployed before thinking about getting a SIEM. Otherwise, you might not need one just yet.

On top of that, a SIEM provides a single mechanism for fine-tuning by using KQL, SPL, or whatever other query language, and that makes exclusions easier to document and manage. If you were to fine-tune downstream, the way you'd do it would be different for every tool, and that requires specialized knowledge.

And finally, you get to learn the new tool as you work on those alerts. And by starting with centralizing alerts you get a chance to get familiar with that new environment while the workload is still “light”.

In your first 30 days you want to do two other things: start working on your list of crown jewels and lay down basic security monitoring processes.

We start discussing crown jewels early because organizations often don't have a clear vision of what theirs are. That gives them at least 60 days (phase 1 + phase 2) to come up with a list.

As far as the processes are concerned, you want to figure out at least three things:

1. [If you're an MSSP] Co-management, or "who does what." This process is your way of managing expectations. People responsible for selling the service sometimes get creative :). You want to establish clarity right off the bat to avoid unpleasant surprises further down the road.

2. Escalation paths: a list of contacts for different technologies. Essentially SMEs who can give you context for investigations and incident response.

3. Knowledge management: a place to keep your procedures. As alerts and data start flowing in, you'll want a central place to manage your knowledge. Can be anything, really, as long as everybody on the team has access and can use it.

That's it. Alerts are flowing in, fine-tuning is taking place, you've established basic processes. You can already start monitoring.

Now you're ready to build detections:

[Phase 2] Build detections (30 days)

The goal of pre-onboarding was to build a list of techniques most likely to be used against you or your client. This phase focuses on creating detections that trigger when those techniques are observed in the environment.

To remind you of what we're going for:

We have threat intel, we’ve ideally drafted detection rules. We’re ready to start onboarding data to our SIEM.

Now, when sending data to a SIEM, we’re met with different data storage tiers. There’s typically some type of hot storage (readily available, also expensive) and cold-er storage (not immediately available, but cheaper). A simple rule to follow is that data that directly supports running detections and investigating them goes to hot storage. What you do with your cold storage is your business, good sir.

Detection engineering workflows can get complex, but if you're just starting out, I suggest you make sure your detections are tested and have a clear SOP. No need for an elaborate detection engineering pipeline at this stage.

The threat intel → detections process should be repeatable. You do it once during onboarding and repeat every quarter or so. This is mostly because the threat landscape continues to evolve, and the detections you've written today may not be the ones you need a year from now.

Lastly, having a structured detection engineering process makes sense, but in reality you will have to write a lot of ad-hoc rules based on ongoing incidents, threat hunts, investigation findings, or just the new attacks that seem to be popping up every other week these days. My approach is this: if there's an urgent need to write a detection, write it. If not, fall back on your detection backlog.

Very simple if you think about it.

Once alerts are in and custom detections have been created, you already have a solid SIEM in place. You could very well choose to finish the work here. Still, I think it makes sense to take the onboarding one step further and aim to protect your crown jewels.

[Phase 3] Protect Crown Jewels (30 days)

This part can get difficult, but bear with me.

We focus on crown jewels last. This is very important because one of the more common SIEM onboarding anti-patterns is setting up monitoring for crown jewels too early.

By the time a threat actor reaches a crown jewel, they're typically late in the attack chain. If you focus on monitoring the most critical assets first, you decrease your ability to detect and stop the attack at earlier stages.

You need to know that companies rarely have a clear concept of crown jewels, and the effort to come up with a list often ends with the security team deciding on their own. Here's roughly the workflow:

However you arrive at the list of crown jewels, make sure to come up with a concise list. This means 3-5 systems, not 50. From that point on, the work seems relatively straightforward - collect data, write detections. And yet, you should be ready to face the following issues:

• Crown jewels are often older systems and may not support data export.

• Clients may hesitate to install agents on crown jewels due to security or performance concerns.

• Maintenance windows may stretch far beyond your onboarding timeline.

• External technicians or vendors might be required for support.

• Crown jewels may not produce useful or relevant data.

If data collection isn't possible, use crown jewels to add context to your existing detections. Automate alert prioritization so that anything related to crown jewels stands out to analysts. Even if you can't pull logs directly, automation can help highlight events that touch crown jewels one way or another.

That's it. You're mostly done. The next steps are the finishing touches:

• Writing simple automation: Teams/Slack notifications for high-severity alerts, automated IOC lookups, tagging of high-value assets.

• Preparing handover documentation.

• Subscribing to SecOps Unpacked.

SIEM onboarding unpacked

Aaaand that’s the 3-phase SIEM onboarding approach, developed through plenty of trial and error. If there is one way you take from this article I'd like it to be the value of relying on threat intelligence to build a SIEM.

Feel free to steal this approach. And if parts of it are not clear - feel even more free to reach out to us on LinkedIn.

Bonus tip for MSSPs

When should you start monitoring?

Most MSSPs I've worked with have a flat fee for SIEM setup and then a recurring fee for security monitoring services. Typically, security monitoring kicks in after the SIEM is fully stood up. This in essence means that you're not paid for monitoring until the SIEM onboarding finishes.

But what if the SIEM onboarding project runs longer than expected? I typically plan 90 days for a full end-to-end onboarding, but I've seen it stretch into years. Many reasons:

• The client's IT team might not be available to assist you.

• Major changes in leadership on the client's end can happen.

• A major incident can occur while you're onboarding.

• A vendor needed to help you might not be available.

• The list of crown jewels can take ages to be prepared.

• Various bureaucracy issues can pop up as you go.

• The client may start disputing the project's scope.

• And so on…

One MSSP I worked with had a nice way around this. They built it into their contracts and called it a hybrid-operations model. What this means is that the SOC team started monitoring as soon as alerts hit the SIEM, which happens in the first phase of onboarding. You don't have to wait for the project to finish to start getting paid.

It accomplishes two important things: it makes handover smoother because analysts have already been working in the SIEM for a couple of months, and it protects your organization against the project running longer than expected. This approach saved us a number of times.

Worth considering.

Check out our SecOps Market Landscape tracker and evaluation frameworks

SecOps Market Landscape

If you have been watching the AI SOC space, you already know what comes right after it. Agentic threat hunting. AI threat hunting. Pick your favorite name, every vendor has one now. It is the second wave, and it is getting loud.

So before we judge it, let me back up and talk about what threat hunting actually is, why people love it, and why automating it is trickier than the demos make it look.

A bit of history

Threat hunting got popular around 2010, back when everyone was chasing APTs. It was the buzzword of the moment, and honestly it was one of the cooler things you could put on your plate as a practitioner. You were not just clearing a queue. You were going after the stuff that slipped past the alerts.

To be a threat hunter you needed two things. First, deep knowledge of your environment. Second, you had to know your SIEM and your query language cold. That second part is where most of the real work lives.

Product Updates !

Main Product Launch

Most security teams cannot tell you which active campaigns their stack would catch today. Not because they lack data. Because converting threat intelligence into validated detections, continuously and at scale, has never been automated. Until now.
Mars connects directly to your SIEM, EDR, identity, and cloud via API. No data ingestion. No tool replacement. It maps active attacker TTPs to your environment and generates production-ready hunts and detections automatically. Your team goes from 5 hunts a month to 50. Built by people who designed attacks like the ones targeting your stack right now.

Introducing Vibe Hunting

Exaforce takes both the hypothesis grind and the query grind off your plate. Threat news articles are ingested automatically as they surface, with IOCs extracted and matched against your environment before any analyst opens a browser tab. And when a hunt does need human intuition, vibe hunting lets analysts express hypotheses in plain language while Exabots handle the querying, correlation, and enrichment.
By the time your CISO asks "are we impacted?", Exaforce has already answered the question

The two kinds of hunts

There are two flavors.

Reactive hunts start from an incident. You have a TTP or an indicator, and you go look for the same activity across the rest of the environment. Did this land anywhere else? How far did it spread?

Proactive hunts start from a hypothesis. Usually that hypothesis comes from threat intel. You read about a technique or an actor, you ask "could this work against us," and then you go check. From there you take it wherever the data leads.

The hardest part in both cases is the same: writing the queries. You need to understand your SIEM, every log source feeding it, and how the data is structured. Then you need to be good at reading and parsing what comes back. That is the skill that takes years to build.

What you get out of it

Outcomes depend on the type of hunt.

Reactive hunts usually give you blast radius. You find out where else the activity showed up, and you spot gaps in your detections or your coverage along the way.

Proactive hunts go one of two ways. Either you find something and it becomes an incident, or you find nothing actionable and the work feeds into detection engineering so they can build new detections. This is why threat hunting and detection engineering are joined at the hip. A hunt that does not feed detections is half a hunt.

So, agentic threat hunting. Is it any good?

Here is the part you came for.

Agentic threat hunting uses LLMs and AI agents to run the hunt for you. And yes, a lot of the hard parts can be automated now. For real, not just on a slide.

An agent can:

• Process threat intel and build a threat profile for your org

• Turn that into hypotheses worth checking

• Write the SIEM (or EDR/NDR/XDR) queries

• Run the hunt and hand you the results

• Suggest coverage gaps and new detection rules off the back of it

For reactive hunts it works the same way. Hand it an indicator and it runs the queries and does the deeper analysis for you.

So the tech is real and it is useful. But the demo is the easy 20 percent. The hard 80 percent is everything that makes a hunt actually work in your environment. Here is what I would push any vendor on.

Question 1: Does it actually know your org?

A hunt is only as good as the context behind it. The agent needs to know your infrastructure, your security stack, your crown jewels, and how you operate. Without that, the hypotheses are generic and the queries point at the wrong things. A hunting agent that does not know your environment is just running someone else's playbook against your logs.

This lines up with a point Anton Chuvakin made at last year's Gartner Security and Risk Management Summit. He put hunting against top-tier nation-state adversaries on the short list of things he does not expect AI to handle any time soon. The easy hunts automate well. The hard ones still need a human who knows the terrain.

Question 2: Is it tuned for your SIEM and your tooling?

This one is non-negotiable. The agent has to be trained and tailored for whatever you run, SIEM, EDR, NDR, XDR. It needs to write correct queries in your query language, not generic pseudo-SQL it picked up off the internet.

From what I have used and built, the thing that separates a good hunting agent from a bad one is simple. Before it writes anything, it should run two quick checks. One, pull the latest log sources. Two, pull the latest schemas. That way it is not guessing at field names, not querying a source you turned off last month, and not burning steps on failed queries.

This is a known failure mode, not me being paranoid. LLMs invent table and field names that sound right but do not exist in your data. Research on enterprise query agents calls this schema hallucination and lists it as one of the main error classes. If the agent does not ground itself in your current schema first, it will hand you a perfectly formatted query against columns you do not have.

Question 3: Can it handle the output without drowning?

Anyone who has written a bad SIEM query knows the pain. You burn your license, you slow the SIEM to a crawl, or you get back a million rows you cannot do anything with.

Agents have all of that plus one more problem: the context window. A bad query that returns a wall of raw logs fills that window fast, and then the agent loses the plot. So the queries cannot just be correct, they have to be efficient. Aggregate before you pull raw events. Select only the fields that matter. Build the query so the SIEM does the stats work and hands back something the agent can actually reason over.

This is not theoretical. Practitioners building these agents already follow the same discipline: run aggregations before pulling raw documents, filter to only the fields you need, use tested query templates instead of building from scratch, and cap how many queries the agent can fire per cycle. The pattern is the same every time. Keep it lean, or the agent chokes.

Question 4: Does it know what normal looks like for you?

Oran Yitzhak pushed me on this one, and he is right. It is not just your stack. It is your behavior. A hypothesis needs a baseline, and the baseline is what is normal in your environment. Without it, the agent produces threats that are technically plausible but operationally meaningless. That is not a hunt. That is a false positive factory with a nicer UI.

This is the part that separates "knows your org" from "knows your tooling." You can wire an agent into the right SIEM with the right schemas and it will still waste your time if it has no sense of what your users, services, and machines do on a normal Tuesday.

The cascading error problem

One more thing worth saying. In a chain like this, mistakes compound. A weak hypothesis leads to a bad query, which gives garbage output, which the agent then reasons over and confidently gets wrong. Each step looks fine on its own. The end result is nonsense delivered with a straight face.

And the confident part is what keeps me up at night. A weak hypothesis does not just produce a bad query. It produces a confident wrong answer. In DFIR a confident wrong answer is worse than no answer at all. It moves people in the wrong direction with conviction, and nobody pushes back, because the AI said so.

One more way to think about it. Agents make good hunters faster. They also make mediocre hunters loudly mediocre. That is useful signal if leadership is watching for it. Most leadership is not watching that metric yet.

That is why the human stays in the loop. Not to write every query, but to sanity-check the hypothesis going in and the verdict coming out. The agent should also show its work: which sources it hit, which queries it ran, what it found and what it did not. "No results" is itself a finding, and a good hunting agent should tell you whether it found nothing or simply could not look.

So where does that leave us?

Agentic threat hunting is real, and it is useful for the repetitive, query-heavy grind. It can take the parts that used to eat your afternoon and give them back. But it is not a hunter in a box. It needs your context, your schemas, tight queries, and a human checking the start and the end of every hunt.

Get those right and it earns its keep. Skip them and you have an expensive agent writing beautiful queries against logs that do not exist.

Same rule as always. It does not replace the hunter. It makes a good hunter faster, and it makes a bad setup fail quicker.

Check out our SecOps Market Landscape tracker and evaluation frameworks

SecOps Market Landscape

Every SecOps platform now has "agents." SOAR vendors, the AI SOC crowd, the new players. Everyone slapped the word on something and shipped it.

But "agent" is doing a lot of work in those sentences. What you actually get changes a lot from one platform to the next. And the differences matter more than the demo makes it look.

At SecOps Unpacked we track this space closely. Right now we count 22 vendors offering some form of agent builder for SecOps. That is a lot of choice, and almost none of them mean the same thing by it.

And even inside that group there is a split. Some platforms ship prebuilt agents and you use them as they are. Plug and play, no customization. That is fine for getting started, but you take what you get. The ones I care about here are the platforms that let you customize those agents or build your own from scratch. That is where the differences show up.

If you read some of my Li post on AI Agents vs. Playbooks, you know I care about where you draw the line between a thing that thinks and a thing that just runs steps. Anthropic draws the same line: workflows run on predefined paths, agents direct their own process and tools. This post is the next layer down. Once you decide you want agents, how the platform lets you build them is the real question.

So let me break down the flavors I keep seeing, and then the harness, which is what actually sets them apart.

Our new Product Updates Section !

Agent Builder release

Most AI SOC demos stop at chat. Fleet is interesting because it gives the you an AI SOC Analyst that does the actual work: reason through evidence, run investigations inside an isolated sandbox, inspect files, execute tools, generate detections, and turn findings into repeatable workflows. The Agent Builder is the control layer on top. Teams can shape specialized agents for package triage, phishing analysis, AIR investigations, reporting, or detection engineering without waiting for a vendor roadmap. This is the kind of AI SOC capability practitioners should evaluate: not autonomy theater, but controlled, customizable execution that maps to how SecOps really works.

Threat Hunting & Threat Intel Analyst

Most threat hunting programs remain out of reach for many security teams because they require specialized expertise and analyst time that is often in short supply. Dropzone new AI Threat Intelligence Analyst and AI Threat Hunter agents aim to make proactive threat discovery more accessible. The Threat Intelligence Analyst monitors threat intelligence sources, extracts TTPs and IOCs, and builds hunt packs, while the Threat Hunter executes them across SIEM, EDR, cloud, and identity platforms. Beyond identifying threats, the agents can uncover misconfigurations, shadow IT, and exposed vulnerabilities, allowing analysts to focus on validation, decision-making, and response.

The Three Flavors

1. The agent you call inside a workflow

This is the classic. Probably the most used today, especially in SOAR and automation-style platforms. You have a workflow, and at some step you drop in an agent to do a piece of reasoning. Triage this alert. Summarize this. Decide if this IOC is bad.

It works. It is easy to reason about, because the agent lives inside a flow you already understand. The blast radius is small. If it does something dumb, it does it in one spot.

The downside is scale. You end up building a new agent per workflow. Phishing flow gets its own agent. EDR flow gets its own agent. Identity flow gets its own agent. Same logic, copied five times, drifting apart over time. And when you want to change how your agents behave, you are editing them one by one. Welcome back to maintenance hell, just with prompts instead of API calls.

2. The monolith agent

One big agent. You call it to do "the reasoning" inside a workflow, and it tries to handle whatever you throw at it.

Honestly I see more downsides than upsides here. The one real upside is ease of use. There is one thing to configure, so getting started is fast. It might also make sense if you are running small or local language models, where you want fewer moving parts to host and control.

But a monolith has no clear role. It is good at a bit of everything and great at nothing. It is hard to test, because the surface is huge. It is hard to guardrail, because it does too much. And when it gets something wrong, good luck figuring out why. You poke at a giant prompt and hope. For a SOC, where you need to trust and audit the decision, that is a bad trade.

3. The agent builder

This is the one I think wins. You build your own agents as reusable units. Each one gets:

• A role. What is this agent for. A phishing triage agent. A host enrichment agent. A containment recommender.

• Constraints. What it is not allowed to do. No isolating critical assets without approval. No closing alerts above a severity.

• A knowledge base. Your SOPs, your asset context, your past decisions. The stuff that makes the agent yours and not a generic model guessing.

• Skills. The actions and tools it can use to get the job done.

Then you reuse them. Build the enrichment agent once, call it from every workflow that needs enrichment. Fix it once, fixed everywhere. This is the same reason we moved from copy-pasted scripts to functions. It is not a new idea. It is just finally showing up in security tooling.

And these get a lot better when they are interactive. An agent you can talk to, that asks for input mid-investigation, that you can correct and steer, beats a fire-and-forget agent buried in a workflow. That is where the co-pilot and the autonomous agent start to merge into something useful.

Now the Harness

Here is what I want you to take away. When you compare agent builders, you are not really comparing models. You are comparing harnesses.

The agent harness is everything wrapped around the model. Tool execution, memory, context management, state, guardrails, the loop that lets it act instead of just answer. The model sits in the middle and does the reasoning. The harness is the rest. And the rest is most of it.

People building production agents keep landing on the same conclusion. The model is the smallest part of the system. When an agent breaks in prod, hallucinates a tool call, repeats an action it already did, ignores an instruction it followed an hour ago, it is almost never the model that got dumber. It is the harness that was underbuilt. Mitchell Hashimoto, the Terraform guy, even named the discipline: harness engineering. Every agent mistake becomes a permanent fix to the environment, not a prompt you retry.

This reframes the whole thing. The agent you call in a workflow, the monolith, the agent builder. These are just different amounts of harness, and different amounts of control over it.

• In the workflow agent, the harness is mostly the workflow itself. You wire context in by hand, step by step. Fine for one flow. Painful across many.

• In the monolith, the harness is hidden inside one big config. You do not really shape it. You feed it and hope.

• In a real agent builder, the role, constraints, knowledge base, skills, memory, and guardrails are the harness. You are configuring it directly. That is the point.

So an agent builder is not a fancy prompt box. A good one is a harness with a UI on top. That framing tells you what to actually look for.

What a Good Harness Gives You in SecOps

When you evaluate one of these platforms, look past the model name on the slide and ask about the harness:

• Tools and integrations as skills. Can your agents reach your SIEM, EDR, IDP, TI, CMDB, and case management, and can you add your own? An agent with no skills is just a chatbot with opinions.

• Memory and context. Does the agent remember prior alerts, prior analyst decisions, known-benign patterns? Or does every alert start from zero. SOC context is the whole game.

• Guardrails as part of the build. Can you set hard limits per agent. Rate limits on anything that quarantines or isolates. Human approval for high-impact actions. This should be config, not vibes.

• Observability. Can you see what the agent did and why. "Selected playbook 42 because 3 of 5 engines flagged the file." If the decision is a black box, you will never trust it, and you should not.

• Evals. Can you test an agent against known cases before it touches prod. Shadow mode counts. Anything that lets you measure before you ship.

• Model-agnostic. This one is underrated. The model is pluggable. A good harness lets you swap it. When a stronger model drops next quarter, and one will, you want to plug it in, not rebuild every agent you own. If your platform welds you to one model, you bought a harness with no spare parts.

Bottom Line

The "agent" label tells you almost nothing. What tells you something is how you build them and how much of the harness you control.

Per-workflow agents are fine to start. Monoliths are easy and not much else. The agent builder, especially an interactive one, is where you get reuse, roles, constraints, and real control.

And the next time a vendor walks in leading with which model powers their agents, ask about the harness instead. That is the part you will actually live with.

Check out our SecOps Market Landscape tracker and evaluation frameworks

SecOps Market Landscape

It has been almost a month since my last post. The reason is simple: I have been heads-down on the new version of the SecOps Unpacked AI for SecOps Vendor Landscape.

The list now has over 110 vendors. A month ago it had around 50. At this rate it could reach 150 by the end of the year.

SecOps Unpacked — Independent Research on AI SOC, Automation & Modern SecOps
Independent research platform for security practitioners. Track AI SOC vendors, security automation, detection engineering, and the frameworks shaping modern SOCs.
SecOps Unpacked • SecOps Unpacked

How this started

The first version was meant to be a simple tracker. Nothing more than a list I maintained for my own reference.

Then practitioners started reaching out. They wanted better filtering. They wanted more depth. They were using the list to scope vendors for evaluation, which was not what I built it for, but it was exactly the kind of use that made me pay attention. Around the same time, engagement on the ARMM framework started climbing, and I noticed SecOps Unpacked showing up near the top of Google results for AI SOC vendor lists.

That last part is what pushed me. If the list was going to rank that well, it needed to be good enough to deserve it. So I decided to take it seriously and build it into a proper app.

Who this is for

Three groups, really.

Practitioners use it to scope vendors for evaluation. You are shortlisting tools, sizing the market, or figuring out which platform to demo or PoC next. This is the primary audience and the reason the list exists.

Investors use it to compare the landscape. If you are doing diligence on a category or a specific vendor, the structured view shows you who is doing what, where the overlaps are, and where the white space is.

Vendors use it for competitive intelligence. If you are building in this space, the landscape is a fast way to see how the market is structured and where you sit relative to everyone else.

The list is free for everyone. No gated content.

I do ask you to subscribe. It keeps you up to date as the list evolves, and it gives me a clearer picture of who is actually reading and using this. That feedback shapes what I build next.

What you get as a practitioner

A structured list of every vendor doing AI for SecOps in some form. I originally wanted to scope it to AI SOC only, but that was too narrow. The market is broader than one category.

To get the structure right, I worked with practitioners I trust: Rafal Kitab, Cristian Miron, and Andrei Cotaie. The category work was also shaped by the writing of Oliver Rochford and Anton Chuvakin, whose thinking on this space is worth reading if you have not already.

What came out of that is a set of categories and definitions covering the full landscape. Each platform is tagged by capability so you can see what it actually does. This is a live list. Capability tagging will be maintained as the market shifts, which is harder than it sounds: for most vendors I need to go through a demo to understand the platform well enough to map it accurately.

Each vendor also has key elements built for evaluation. Easier to see by browsing than to describe here.

I added several visualizations, because that is how I like to consume data. If you have an idea for a visualization that would help, tell me.

Advanced filtering is in the backlog. I am holding it until more vendors claim their profiles and the data is rich enough to make filtering worthwhile.

Why not just scrape vendor websites

Some people have asked why I do not just scrape vendor sites and auto-generate profiles. It is not that simple, at least not the way I want to do it.

I want vendors to share what they think is worth highlighting. Every profile goes through a review where I add editorial notes. In a future version you will be able to see which vendors I have seen a live demo of and which I have not, so you know how much weight to put on each profile.

What is coming

A few things on the roadmap:

An evaluation layer, in two forms. One connected to the ARMM framework. One built as a standalone way to test and evaluate AI SOC vendors, closer to a golden-standard benchmark. This needs a solid plan and the right partners to build the infrastructure around it, so it will take time.

A product release feed, plus a dedicated newsletter for anyone who wants vendor product updates in one place, separate from the main newsletter.

The longer-term goal is to turn this into the evaluation platform for SecOps. Not just a list. A place where you can actually evaluate.

For vendors

Vendors can claim their profile and start filling in information about their company and product. There is also an option for full access, which is subscription based. That covers the maintenance work each profile needs and a few additional perks.

If you are a vendor, claim your profile and I will share the details on the full offering.

So far over 20 vendors have claimed their profile, and over 10 have joined as founding members to support the development of the platform. That early support means a lot and it is what makes the bigger roadmap possible.

That is it for now

If you found this useful, share it. The more practitioners who use the landscape, the better it gets for everyone.

Check out our SecOps Market Landscape tracker and evaluation frameworks

SecOps Market Landscape

Today I want to break down something that keeps coming up in conversations: the difference between SecOps Agents and AI SOC when they live inside a SIEM versus when they show up as a standalone product or as part of a SOAR.

Three different starting points, three different sets of trade-offs. None of them are wrong. They just optimize for different things, and if you don't understand which one you are buying, you will end up disappointed.

SIEMs own the data, and that matters

SIEMs have the upper hand on triage. Full stop.

If you are a team that sends everything to your SIEM, you will get great triage results inside it. The reason is simple. The SIEM has all the data. If the implementation is done right, the platform can query that data faster and more efficiently than anything bolted on from the outside. Timeline analysis, blast radius, correlation across sources, all the bells and whistles.

But there is a condition attached to this. ONLY if you actually get all the data there. And by data I do not just mean logs. I mean enrichments too. Identity context, asset context, threat intel, business criticality, ownership. The triage is only as good as the context the SIEM can reach without leaving its own walls.

This is where the story gets complicated.

Once you move past triage and into response, the SIEM starts to lose ground. SIEM vendors were smart to play the SOAR card years ago. The problem is that most of them never invested enough to make it shine. Acquired SOAR products got bolted on, kept on life support, and then quietly underfunded while the marketing kept going.

My take has not changed. Unless you have your entire response stack inside a single ecosystem, and very few teams actually do, you need a vendor agnostic agentic, automation, and orchestration layer. The SIEM is a great triage brain. It is rarely a great hands and feet.

Where pure play AI SOC and SOAR have the upper hand

Now flip the scenario. You don't send everything to your SIEM. Maybe you cannot afford to. Maybe you are running multiple detection sources. Maybe your EDR, your cloud detections, your identity alerts, and your email security all live in their own consoles.

This is where pure play AI SOC vendors and SOAR-style platforms shine.

They do enrichment better in this world because they were built for a fragmented data reality. They pull from wherever the data lives instead of assuming it all flows into one lake. The trade-off is that they are limited by whatever APIs are available on the SIEM and on every other system they ingest alerts and detections from. If the API is shallow, the agent is shallow.

On the response side, this is where SOAR-style and agentic platforms really pull ahead. Response is hard. It involves dozens of systems, conditional logic, human approvals, rollback paths, and edge cases that only show up in production. Slapping MCP on top of a tool catalog and calling it response automation does not make the cut. Response needs to be designed, not summoned.

TL;DR for the architecture decision

If you have all your data and all your response actions inside one ecosystem, ride the SIEM agent wave. It will probably work for you.

If you do not, and most teams do not, you need a layer that is vendor agnostic on both data and response. That is where pure play AI SOC and modern SOAR-style platforms earn their keep.

Vendor tracker update

A quick update on the AI SOC and Agentic SOC vendor list we maintain.

We are now tracking 73 vendors. The market keeps expanding, and we keep adding entrants as they show up with credible product and not just a landing page.

A few changes worth calling out:

• We now have multiple visual views of the landscape. Different cuts for different questions. One view for category, one for go-to-market motion, one for how they handle the data and response split discussed above.

• The next iteration will include a definition for each category. One thing I have learned from talking to practitioners and buyers is that the category names are doing a lot of heavy lifting and not always carrying the weight. AI SOC, Agentic SOC, autonomous SOC, alert triage copilot, they all mean different things to different vendors. We will pin down what each one actually means in our taxonomy so the comparison is honest.

If you want to be on the list, or if you think we missed you, reach out. The bar is product that exists and customers who use it.

Coming up: AI SOC is just a feature

Last thing. I will be joining Chris Hughes next week on Resilient Cyber, and one of the topics I want to dig into is why I think AI SOC is just a feature, not a category.

Short version of the argument. Triage automation, alert summarization, investigation assistance, these are capabilities. They will be embedded in SIEMs, in SOAR, in EDR, in detection engineering tools. Standalone AI SOC vendors that do not extend into the rest of the SecOps lifecycle will get squeezed. The interesting companies are the ones building agentic platforms that go beyond the triage box.

More on that in the episode.

Check out our SecOps Market Landscape tracker and evaluation frameworks

SecOps Market Landscape

If you've been following this blog, you know the story so far. 2024 gave us AI copilots that summarized alerts but took no action. 2025 proved AI could actually triage, investigate, and reach verdicts at machine speed. The "AI Analyst" became a real product category.

My take: we automated the easy part.

The industry started in the middle of the IR cycle. Triage is analytically complex but operationally simple. No write access required. No change management. No risk of breaking production. It was the perfect first target. Detection engineering sits left of that, tangled in log pipelines and alert volume constraints. Response sits right, blocked by API limitations and organizational risk aversion.

And now we have a new problem.

The Decision Load Tripled

Faster detection and better triage surface more decisions that humans need to make. Before AI, a SOC might process 200 alerts daily and make 50 meaningful decisions. Now AI surfaces 2,000 alerts, auto-closes 1,700, and escalates 300 that require human judgment.

Detection speed improved. Decision velocity did not.

If you read my post on how AI transforms detection engineering, you know I talked about the shift from precision-optimized detection to coverage-optimized detection. The idea is simple: when AI handles triage at scale, you can deploy the detections you always wanted. Broader rules, more coverage, let the AI sort through the noise.

That's true. But it created a side effect nobody planned for.

More coverage means more signals. More signals means more escalations. More escalations means more human decisions. And if those detections are noisy, poorly tuned, or written under the "Fear of Not Doing Enough" (yeah, that post still haunts me), your AI SOC is just processing garbage faster.

Just because AI can investigate and triage alerts faster doesn't mean we should feed it bad detections. We don't want noise detections burning through tokens the same way we didn't want noise detections burning through SIEM licenses. Different cost center, same problem.

Think about it. With traditional SOAR, bad detections cost you analyst hours. With AI SOC, bad detections cost you tokens, compute, and worst of all, they erode trust in the system. Your analysts see the AI confidently closing garbage alerts and start questioning whether it's also confidently closing real threats. That's how you kill adoption.

2026 Will Test the Shift Left

This year will test whether AI can shift left into detection and shift right into response. Not just reorganize the human decision burden but actually reduce it.

I think the shift left is quite important right now. And there are two ways to approach it.

Path 1: The Feedback Loop (Your AI SOC Should Pay for Itself)

If your AI SOC is not giving you suggestions on how to improve your detections, it's designed to charge you more. Full stop.

I've been preaching about feedback loops since the DSAEM days. Detection > SOP > Automation > Emulation > Metrics. The loop is what makes the whole thing work. Without it, you're just driving faster into a wall.

I don't think the AI should give you a one-to-one suggestion for every single alert. That's noise on top of noise. What I see working better is a periodic review. Weekly or monthly. The AI goes over your alerts and incidents, runs analytics across the dataset, and comes back with suggestions.

Things like:

• "This detection fired 847 times last month. 812 were auto-closed as benign. Here's a filter that would eliminate 90% of the noise without reducing true positive coverage."

• "These three detections overlap significantly. You could merge them into one with better context and reduce your alert volume by 30%."

• "This detection has a 2% true positive rate. Either tune it or kill it."

• "Based on analyst overrides, your AI is consistently wrong about this alert type. Here's what the analysts are seeing that the AI is missing."

That last one is gold. When analysts override AI decisions, that's training data. Not just for the AI model, but for your detection engineering program. If the AI keeps getting a specific detection wrong, maybe the detection itself is the problem.

The feedback loop isn't just about making the AI smarter. It's about making your detections smarter. The AI sees patterns across thousands of alerts that no human analyst has time to analyze. Use that.

If your AI SOC vendor doesn't offer this, ask why. Because from where I stand, an AI SOC that doesn't feed back into detection improvement is just an expensive alert processor. And we already had one of those. It was called SOAR.

Path 2: Building Better Detections from Scratch

The feedback loop fixes existing detections. But what about new ones? How do you build better detections from the start?

This is the harder problem. And it's where most teams are still stuck in manual mode.

Let me break down how I think about it.

Where New Detections Come From

In most organizations, new detections come from two places:

Threat Intelligence. You get a feed. IOCs, TTPs, reports about new attack techniques. That intel should feed into your Threat Profile and Threat Modeling. Not every piece of intel is relevant to you. The question is always: does this threat apply to my environment, my industry, my infrastructure?

Threat Hunting. Your hunters go looking for things your detections missed. When they find something, that finding should become a detection. In many cases, threat hunting is also TI-led. The hunter reads a report about a new technique, goes looking for it in the environment, and either confirms or denies its presence.

So the flow looks like this: Threat Intel feeds into both Threat Modeling and Threat Hunting. The outcomes of those processes produce detection suggestions.

Simple enough on paper. In practice? Total mess.

Here's where it falls apart

You get a detection suggestion. Maybe it's a Sigma rule from a community feed. Maybe your hunter wrote it after finding something interesting. The question is: should I implement this?

To answer that, you need to know:

• What log sources do I actually have?

• What infrastructure am I running?

• Do I even have the telemetry to detect this technique?

• If I implement this detection, what coverage does it give me?

• What's the gap if I don't implement it?

Most teams answer these questions from memory. Or they don't answer them at all. They just implement the detection and hope for the best. That's the "Fear of Not Doing Enough" in action. Write the detection, push it to production, deal with the consequences later.

This Is Where AI Should Help

An AI-powered or agentic detection engineering platform should handle exactly this workflow:

1. Ingest your threat intelligence. Not just IOCs. TTPs, actor profiles, campaign reports. Map them to your threat model automatically.

2. Understand your environment. Know what log sources you have, what infrastructure you're running, what telemetry is available. This is the foundation. Without it, everything else is guessing.

3. Match threats to coverage. Based on your threat profile and your available telemetry, show me what I can detect and what I can't. Where are the gaps? What's the risk of those gaps?

4. Suggest detections that make sense. Not generic Sigma rules dumped into a folder. Detections that are relevant to my environment, my log sources, my infrastructure. With the context of what coverage they'll add and what risk they'll reduce.

5. Measure everything. Coverage mapping, risk exposure, detection health. The whole picture.

What to Measure

Since we all love our MITRE ATT&CK bingo cards (don't pretend you don't), coverage mapping is the obvious starting point. But it can't stop there.

Coverage. Map your detections against ATT&CK techniques. Show me what's covered and what's not. Yes, it's bingo. But it's useful bingo when you combine it with the next piece.

Exposure and Risk. For every gap in coverage, I need a way to calculate risk. Not some abstract risk score pulled from thin air. Something grounded in: what threat actors target my industry? What techniques do they use? Do I have compensating controls? What's assumed risk versus mitigated risk?

This is how you go from "we have 60% ATT&CK coverage" to "we have 60% coverage, and the 40% we're missing exposes us to these specific attack paths, with this estimated risk, and here's what we'd need to close those gaps."

That's a conversation a CISO can actually use. Not another bingo card.

Detection Health. Are your existing detections still working? Are they firing? Are they producing true positives? Or have they degraded because the environment changed and nobody updated the rule? This ties back to the feedback loop. A detection you wrote six months ago might be useless today if the infrastructure changed.

Connecting the Dots

If you zoom out, the picture looks like this:

The AI SOC handles triage and investigation. It's fast, it's scalable, it works. But it only works as well as the detections feeding it. Garbage in, AI-powered garbage out.

The feedback loop takes what the AI learns during triage and feeds it back into detection engineering. It's continuous improvement. Your detections get better over time because you're learning from thousands of alert outcomes, not just the handful an analyst has time to review.

Building better detections from scratch takes the proactive approach. Instead of waiting for bad detections to generate noise, you start from threat intelligence and threat modeling. You map coverage, identify gaps, calculate risk, and build detections that actually matter for your specific environment.

The two paths complement each other. One fixes what you have. The other builds what you need.

And this is exactly the shift left that 2026 needs. We proved AI can triage. We proved it can investigate. Now we need to prove it can help us build better defenses from the start.

Vendor Spotlight: Spectrum Security

First platform tackling the agentic detection engineering problem. Details on how they approach the TI > Threat Model > Coverage > Risk workflow, their coverage mapping capabilities, and how they connect threat intelligence to actionable detection suggestions based on your actual environment and log sources.

Spectrum Security is tackling the part of the problem most of the market has worked around for years: detection itself.

While much of the industry focused on collecting more data, shipping more content, or accelerating triage, Spectrum starts from a more fundamental question: can you actually detect the threats that matter in your environment right now? That is the question their platform is built to answer.

The company’s thesis is that security teams do not suffer from a lack of telemetry. They suffer from inability to detect threats through all the telemetry - it’s like searching for a needle in a haystack. More logs do not automatically create coverage. More detections do not automatically reduce exposure. It’s about the need to have the right detections needed for your environment, and ensuring they remain effective as threats evolve and environments change. static dashboards or ATT&CK heat maps do not provide continuous confidence in a changing environment.

Spectrum is building around that gap.

Their approach connects the pieces security teams have historically managed in separate systems and spreadsheets: measuring and reporting on current coverage ,threat intelligence, threat relevance, telemetry availability, detection logic, detection authoring, and ongoing validation. In practice, that means taking threat information and mapping it to the customer’s real environment, understanding what data is actually available, identifying what is detectable versus what is not, and turning that into accurate detections, ensuring that they remain valid and tying this to measurable coverage outcomes. This is meant to replace manually-heavy detection engineering that struggles to keep pace, with something more continuous, contextual, and provable.

What makes the vision interesting is that Spectrum is not describing detection as a one-time engineering project. It is describing it as a living system. Environments change. Telemetry shifts. Threats evolve. Detection logic drifts. So the real problem is not just creating detections, but continuously knowing whether they still work, where gaps have opened, and what matters to fix next. That is the layer Spectrum…

If the first wave of AI in security helped analysts move faster once alerts that already existed, Spectrum is betting the more important shift is upstream: helping teams know what they should detect, what they can detect, and how to close the gap between the two

Final Thoughts

We spent the last two years optimizing the middle of the IR cycle. Investigation and triage are faster than ever. That's good progress. But faster triage doesn't fix bad detections. It just processes them quicker.

The shift left into detection engineering is where the real value is. Not because it's easy. It's actually the hardest part. But because everything downstream depends on it. Your AI SOC, your response automation, your coverage, your risk posture. All of it starts with whether you're detecting the right things in the first place.

Fix the input. The output takes care of itself.

Check out our SecOps Market Landscape tracker and evaluation frameworks

SecOps Market Landscape

Who Signs Off? AI, The Accountability Anchor and the "Basic" Problem

You have two choices: you either surf the AI wave, or you get caught under it. Surfacing is possible, but the landscape will be unrecognizable.

The current "AI freak-out" isn't actually about Large Language Models (LLMs) becoming sentient. It’s about the displacement of agency. We are moving from a world where humans make every granular call to a delegated model where the Accountability Anchor is the only thing keeping the ship from drifting away. We’ve seen these waves before-from the Luddites smashing looms in 1810 to the SOAR hype that promised the "death of the SOC" a few years ago.

But this time, the wave is massive, powered by a projected $2.52 trillion in global spending by 2026. If we don't build a floor to elevate the workforce, we risk installing a ceiling of obsolescence similar to the "Basic Assistance" trap on Earth in The Expanse series.

The ROI Reality Check: Why the Wave is Rising

Businesses aren't adopting AI because they love the technology; they are doing it because they have to prove value through operational cost reduction and profit maximization. While some companies will take out humans from their jobs just to show a reduction in costs, the smart ones see it as a performance multiplier.

But the threat landscape isn't waiting for anyone to figure out their AI strategy. According to Mandiant's M-Trends 2026 report, based on over 500,000 hours of incident response engagements:

• Speed kills, literally: The median time between an initial access partner breaching an environment and handing off access to a secondary threat group (often ransomware operators) has collapsed to 22 seconds in 2025. Down from over 8 hours in 2022. That's not a typo. Alerts traditionally considered "lower priority" can become full-blown ransomware incidents before a human finishes reading the ticket.

• Dwell time is climbing, not shrinking: Global median dwell time rose to 14 days (from 11 in 2024), driven by espionage operations and DPRK IT worker campaigns averaging 122 days of undetected presence. The idea that we're getting better at detection across the board doesn't hold up when state-sponsored actors are living in your environment for four months.

• Recovery is the new target: Ransomware operators have shifted their primary objective from data theft to deliberate recovery denial, systematically destroying backup infrastructure, identity services, and virtualization management planes. They're not just encrypting your data. They're making sure you can't get it back without paying.

• The spending follows the pain: Average monthly AI budgets are jumping from $62,964 in 2024 to an estimated $85,521 in 2025, a 36% increase. Organizations using AI-powered security platforms report identifying breaches significantly faster and reducing average breach costs by roughly 43%.

When the hand-off from initial access to ransomware deployment happens in 22 seconds, no human is reviewing that alert in time. Defenders are forced to automate just to stay in the game.

The Accountability Anchor: Why Humans Still Own the Risk

In the short to medium term, security jobs are anchored by the legal and ethical need for a human to "sign off." Even as we automate the labor of investigation, the Accountability Anchor ensures that responsibility doesn't vanish into a "black box" where no individual bears the consequences of a failure.

Without human oversight, purely automated decisions can create "accountability vacuums." In high-risk environments like cybersecurity, technology excels at velocity and pattern detection, but it lacks the human capacity for context and consequence.

The Circuit Breaker Problem

The industry has been selling Human-in-the-Loop (HITL) as the safety mechanism that keeps AI in check. The idea is simple: define where the algorithm's authority ends and human discretion begins, force the system to pause at critical moments, hand control to a person who can navigate ambiguity. In theory, this acts as a "circuit breaker."

In practice? The circuit breaker is mostly decorative.

Anthropic recently published data on how users interact with Claude Code permission prompts. The numbers are hard to ignore: 93% of permission prompts get approved, and Anthropic themselves describe this as "approval fatigue, where people stop paying close attention to what they're approving." New users with fewer than 50 sessions auto-approve about 20% of the time. By 750 sessions, that number climbs past 40%.

This isn't unique to coding agents. It's a pattern anyone who has worked in a SOC will recognize instantly. Alert fatigue. Approval fatigue. It's the same cognitive failure mode wearing different clothes. When you ask humans to approve hundreds of actions per day, they stop reading and start clicking.

And the threat landscape is evolving specifically to exploit this gap. M-Trends 2026 shows voice phishing (vishing) jumped to the second most common initial infection vector at 11%, while traditional email phishing dropped to 6%. Attackers aren't sending bulk emails anymore. They're calling people, building rapport in real-time, and exploiting the human tendency to trust a live conversation. The initial infection vectors are getting more human-targeted at the exact moment we're asking humans to be the safety control for AI systems.

The data also reveals something more nuanced than "nobody pays attention." Experienced users don't just approve more; they also interrupt more often. New users review each action upfront and rarely intervene (about 5% of turns). Experienced users let the agent run and step in when something goes wrong (about 9% of turns). This is a deliberate shift from proactive per-action review to reactive monitoring and intervention.

This distinction matters. Per-action approval is not a security control. It's a ritual. The real oversight is happening when experienced operators watch the system's behavior, recognize drift, and pull the emergency brake at the right moment. That is the actual circuit breaker, and it looks nothing like a "click approve" dialog.

When the Anchor Becomes a Rubber Stamp

Here's the uncomfortable question: if the person who is supposed to "sign off" is approving 93% of the time without meaningful review, do you still have accountability? Or do you have compliance theater?

Anthropic's own incident log provides a clear answer. Real-world agentic misbehaviors they've documented include agents deleting remote git branches from vague instructions, uploading an engineer's GitHub authentication token to an internal compute cluster, and attempting migrations against a production database. These are not hypothetical "what if" scenarios. These are things that happened because an agent acted and a human either wasn't watching or clicked "approve" without reading.

The honest answer is that accountability needs to evolve. It can't live at the per-decision approval layer because that layer is broken at scale. The Accountability Anchor needs to move up the stack: the person who answers to the board or the regulator isn't clicking "approve" on every alert closure. They are accountable for ensuring the automation is trustworthy, bounded, and auditable. They own the system design, the guardrails, and the audit trail. Not the individual clicks.

This means building hard boundaries into infrastructure: explicit trust boundaries, tool permissions, action constraints at the architecture level. What environments can the agent access? What actions can it take? What data can it touch? These decisions should be baked into the agent's configuration and enforced programmatically, not left to runtime approval prompts that data shows will get rubber-stamped the vast majority of the time.

M-Trends 2026 reinforces this point from the attacker's side. Ransomware operators are now systematically targeting backup infrastructure, identity services, and virtualization management planes before deploying ransomware. They're not just encrypting your production environment; they're destroying your ability to recover. If your "accountability" layer is a human clicking approve on alert closures, you've already lost. The guardrails need to be baked into the architecture itself: immutable backups, identity isolation, hardened recovery paths. The same principle applies to AI agent governance. Don't rely on the human click. Build the boundaries into the system.

The Responsibility Map

The real gap isn't technology. It's the accountability layer. You can automate the triage, but you cannot automate the person who answers to the board or the regulator when things go wrong. What you can (and should) automate is everything below that person's decision threshold, with guardrails that actually work instead of approval prompts that don't.

ARMM: Evolving Beyond the "Vibe Check"

To move beyond "vibe adoption," we need a maturity model. The AI Response Maturity Model (ARMM), developed by Andrei Cotaie, Cristian Valeriu Miron, and Filip Stojkovski, provides that path.

Maturity isn't just about having an AI; it's about scoring it on three axes:

1. Trust: Do you trust the output enough to let it act?

2. Complexity: Can your team actually maintain this model?

3. Impact: What is the "blast radius" if the AI fails?

What's interesting is that Anthropic's behavioral data validates this progression in the wild. Users naturally move through ARMM levels as they gain experience:

• At Level 2 (AI Assistance), the bot suggests and the human approves every action. This is where new users start, reviewing each step before execution.

• At Level 3 (AI Collaboration), the human shifts to monitoring and intervention. This maps directly to what Anthropic observed: experienced users let the agent run autonomously and interrupt when something drifts.

• At Level 4 (AI Delegation), specialized agents act independently within defined bounds. This is where hard boundaries, deterministic controls, and infrastructure-level guardrails become non-negotiable, because the human is no longer in the per-action loop at all.

The ARMM model helps us evolve roles rather than delete them. But it also forces an honest conversation: if you're claiming to operate at Level 2 while your users are behaviorally operating at Level 3 or 4 (approving everything, monitoring from a distance), you have a maturity gap disguised as a process. Fix the process to match reality, or reality will fix it for you.

The "I, Robot" Shift: When AI Gets a Body

The true long-term impact on the job market isn't just driven by code, but by the "Physical Turn" the convergence of AI and humanoid robotics. This is where the I, Robot vibe becomes a business reality.

• The Scale: Experts project there could be over 1 billion humanoids on Earth by 2050 to offset global labor shortages.

• The Price Tag: While a humanoid cost $200,000 in 2024, costs are expected to drop to $13,000–$20,000 by the early 2030s.

• The Tech: Breakthroughs in "Vision-Language-Action" (VLA) models allow these machines to learn and adapt to unstructured human environments rather than just following a script.

While this sounds like science fiction, it raises the accountability bar to its highest level. A hallucinating chatbot writes a bad email; a hallucinating humanoid has a real-world "blast radius."

The Expanse Metaphor: Floors vs. Ceilings

In the series The Expanse, Earth has a population of 30 billion, but only half have jobs. The rest live on "Basic Assistance" -free food, free housing, and recycled paper clothes, but zero money and zero opportunity.

"Basic" isn't a floor; it's a ceiling. It’s a way to manage a population rendered "obsolete" by automation. As we evolve roles like the Tier 1 SOC analyst, we must ensure we aren't removing the "stepping stones" for new talent. If we automate the path to expertise, we end up with a future of job scarcity where only the "proven" get to work.

We must decide if AI will be used to lift everyone above a "poverty floor" through Universal Basic Income (UBI), or if it will be used to construct a "Basic" ceiling that traps the majority of the population in a state of manufactured scarcity.

Leveling Up: Your Career Anchors

The Tier 1 SOC analyst role isn't disappearing. It's leveling up. The manual grunt work of copying and pasting IPs is being replaced by strategic roles. And the data supports the shift: M-Trends 2026 shows 52% of compromises are now detected internally (up from 43% in 2024), which means organizations investing in detection capability and internal tooling are seeing results. The roles driving that improvement:

• Detection Engineer: Designing the behavior-based models that the AI runs.

• AI Validation Specialist: The person who "validates the autopilot" before the plane takes off.

• Governance Officer: Owning the accountability layer between the silicon and the board.

The three of us all started in the trenches of the SOC. We aren't there now because we evolved with the technology, and the industry is doing the same.

The real question isn't whether AI will take your job-it's whether you'll be the person who owns the risk when it does. But until then, let us thank the heavens for accountability.

We apologize if this felt like a mission briefing for the Rocinante, but Andrei Cotaie is a massive fan of The Expanse and we couldn't stop him from geeking out over the "Basic" problem.

Sources and Further Reading

• Anthropic, "Measuring AI Agent Autonomy in Practice" (2026). The research behind the 93% approval rate, behavioral patterns of experienced vs. new users, and the shift from per-action approval to monitoring-and-intervention.

• Mandiant / Google Threat Intelligence Group, "M-Trends 2026" (2026). Based on 500,000+ hours of incident response engagements in 2025. Source for the 22-second hand-off metric, 14-day global median dwell time, vishing as #2 infection vector, recovery denial trends, and 52% internal detection rate.

• Chris Hughes, "The Human-in-the-Loop Illusion," Resilient Cyber (2026). A complementary analysis of the HITL problem and Auto Mode implications, including Simon Willison's critique on non-deterministic AI safety controls and the UK AISI data on agentic tool growth.

Check out our SecOps Market Landscape tracker and evaluation frameworks

SecOps Market Landscape

Back from RSAC, still jet lagged. But I need to get this out while the thoughts are fresh.

I have been going to RSAC for years as a practitioner. This year I went as both practitioner and vendor. That changes everything. Running a booth, doing over 20 demos, having 50+ conversations just around the booth. To speak as a vendor I needed to reverse engineer what I was doing as a practitioner, what I wanted to hear, how things needed to be explained so they resonate with everyone. Seeing the conference from both sides gave me a perspective I did not have before, and I think it made this one of the best RSACs I have attended.

More about the vendor experience in a dedicated blog on BlinkOps. Here I want to share from the practitioner lens. What I saw, what got validated, and what the industry is still getting wrong.

The Conference Itself

The quality of talks this year was solid. I have not been on the talk tracks since 2020 when I gave a talk on Intelligent Threat Intel: Lead Framework, so it was good to be back in the sessions.

My favorite was Anton Chuvakin's STR-W08: Shadow Agents: A Pragmatist's Guide to Governing Unsanctioned AI. It triggered some thoughts on how to build better governance around agents. I already drafted a high level design and will release it soon. What made it even better is that some of the things we built at BlinkOps are aligned with how you can actually do better governance for agents and how you can organize them more efficiently.

Beyond the talks, I had a ton of meetings and caught up with a lot of peers. Amazing conversations, great ideas exchanged.

The Uncomfortable Truth About AI Adoption

One theme came up over and over in those conversations. In the past we were able to build tech and say with confidence this is the future and it will be used in the next 3-5 years. Now it is hard to predict how the tech will look in 1 year, all due to the pace of developments around AI and agents.

But here is the interesting part. Not many are at the stage where they implement it at scale. The majority are using AI mainly around copilots. Agentic implementations are still early days and getting traction only between early adopters.

The main issue persists: adoption is messy. It is like we learned nothing from moving from on-prem to the cloud.

A Quick Note on Booth Culture

I will keep this short because I could rant. As usual the floor was full of shiny distracting booths that make you feel like you are in a theme park. If I can't understand what you are selling from a single sentence at the booth, for me that is a no-go. Your booth can have the most awesome visuals and bring cool attractions, but if the messaging is lost then what is the point.

Interesting enough, the smaller booths had better messaging than the large ones. Many of the big ones would just display the vendor name with no context(this works for large well known brands). If I was not stopping to get scanned and watch a demo I had no idea what their product does. That tells me they are not aware of their own brand awareness gap.

AI SOC Is Not a Product, It Is a Feature

This is probably my hottest take from the show, so let me just say it clearly.

AI SOC or Agentic SOC is not a product. It is a feature.

What I saw on the floor is that AI SOC has become a core functionality embedded in many platforms. Almost every SIEM (if not every) now has some form of AI SOC capability. Some are better than others, some are more of a checkbox exercise. But the autonomous triage and base analysis that was initially pitched as a standalone product category is becoming just another feature layer.

On top of that, SIEM vendors all started adding the response (SOAR) piece as well. Elastic for instance announced their Automation capability.

What I heard asked quite often around AI SOC was: How can you give feedback? How does it learn from past experience? Can it be customized? And what else beyond triage can it do?

Those are the right questions. And the vendors that can answer them well will be the ones that survive the consolidation wave.

Where Most AI SOC Vendors Are Falling Short: Response

Now here is where it gets real. The area where I see most AI SOC vendors struggle is response.

Getting a UI where you can build automations is not going to cut it. And doing the lazy route of saying you can connect to MCP and call it a response layer is not going to cut it either. MCP is a protocol not a response strategy.

To do response properly you need a solid integration layer with deep connections into the tools your SOC actually uses. You need orchestration logic, error handling, feedback loops, and many other components that make the difference between a demo and a production deployment. And yes, to be truly functional in this space you need to be able to build agents. Not just use them, build them. That is where the real differentiation lives.

Triage is getting commoditized. Response is where the hard problems are. And most vendors are not there yet.

Predictions Validated

For me RSAC was also a validation moment. Seeing my predictions play out on the floor is a testament that the analysis work we do at SecOps Unpacked holds up.

Prediction 1: AI SOC becomes a feature. Covered above. It is happening across the board.

Prediction 2: AI SOC acquisitions start this year. Not one but two happened already. Culminate was acquired by Datadog. Kenzo Security was acquired by Rapid7. More will come. My bet is we will see at least 5 AI SOC vendors acquired this year.

Prediction 3: AI SOC vendors shift towards MDR or detection engineering. And that is exactly what is happening. The ones that don't go the MDR route are pivoting towards detection engineering. Two clear lanes forming.

Shadow IT and Shadow AI Are Not Going Away

Another topic that persisted through RSAC was Shadow IT and Shadow AI. I think the vendors that will have the most success going forward will be platforms that can govern both shadow IT and shadow AI. The components needed are identities combined with MDM, SSE, ZTNA, and DLP. That is a lot of tech converging in one place, and it probably deserves its own in-depth blog.

Cool Tech Worth Watching

I want to give credit to some teams building impressive things that caught my attention during the show.

Spectrum Security - Finally had a chance to meet with the founding team, Dylan and John Meny. What they are building is super cool and I think it is the kind of tech that could reinvent how we do threat detection.

Above Security - Aviv and his team have some really interesting tech around insider risk. Huge potential there.

Alpha Level - Joshua Neil and his team are building alert management done right. I really like their approach of combining ML and LLM in a smart way. I think this is how you build real IP around AI SOC.

Tracebit - Andy Smith and Sam Cox are building really cool deception technology platfrom.

And yes, for anything else you need BlinkOps. 😀

Check out our SecOps Market Landscape tracker and evaluation frameworks

SecOps Market Landscape

Note: This is Part 1 of two series around AI influence on the SecOps job market

A Brief History of Freaking Out

The Luddites smashed textile machines in the 1810s because they feared losing their livelihoods. Over 200 years later, we're having the same conversation. Not about looms, but about LLMs.

The fear isn't that somebody will replace you in an org chart. It's that the thing you trained for, the thing you're good at, the thing that pays your rent, the thing that is, ultimately, a part of you might become irrelevant.

When we started working in SOCs, a big part of the day was checking indicators across multiple databases. Copy an IP, paste it into five different platforms, check if something shows up, document the results. Repeat. If somebody asked us today whether that job should still exist, we'd all say no. It makes no sense doing that manually when integrations and automations handle it in seconds.

But here's where it gets personal. For us in 2026, losing relevance could mean losing income. The average millennial should expect to change careers five or six times. Some of us are already on that path. But the question is whether we'll be switching by choice or because the tech forced our hand.

This question got a lot more real a few weeks ago when Block (the company behind Square and Cash App) cut 40% of its workforce, over 4,000 people. CEO Jack Dorsey said straight up that "intelligence tools have changed what it means to build and run a company" and predicted most companies would follow within a year. Investors loved it. The stock jumped 22%. For the people who lost their jobs, less exciting.

Now, whether Block's move was truly AI-driven or just pandemic overhiring correction with an AI narrative (there's strong evidence for both), the signal it sent was loud. And it made a lot of people in our industry nervous.

So the question that needs asking: will the jobs that currently define us become irrelevant? Will our industry reinvent itself? Will we survive this wave the same way we survived SOAR's promise that "the SOC is dead"?

We think the answer is more optimistic than the anxiety suggests, but it comes with some conditions.

The Short Answer

Yes, we will survive this wave.

Why? We'll walk through three arguments and stress-test each one. Some of this thinking was shaped by a talk at Apres Cyber Slopes Summit that helped cut through the noise and get back grounded.

Our Industry Is Different

We know, we know. Every industry says this. Every person who ever worked a job says "nobody can automate MY job because of the unknowns I face every day." My job requires intuition. My job requires split-second decisions. A system can't do that.

All of those statements are simultaneously true and false. But security actually has something unique going for it: the offense-defense arms race.

Security was, is, and always will be guided by a simple philosophy: every new defense technology will ignite the spark to create a new weapon.

We're already at the point where both sides use AI. Defenders have AI SOC solutions that automate investigations, speed up detection, ingest new log sources, and discover anomalies. Attackers didn't waste any time either. Every step of the kill chain has been supercharged since LLMs took over. Script kiddies who couldn't do enough harm before are now targeting higher-value assets with better tooling.

Let's take this to the extreme. Say everybody builds a perfect AI SOC that detects everything from day one. What happens the next day? Somebody builds something that manipulates that system into believing their activity isn't worth alerting on. Even in the most automated scenario, somebody will constantly need to detect, train, and alert on new attack patterns. And someone on the other side will keep finding ways to avoid detection.

That cycle doesn't end. It hasn't ended in thousands of years of warfare, and it won't end because we have better chatbots.

Organisational Resistance to Change

The market will force change eventually. But can you imagine the entire population of CISOs saying "yes, please cut 50% of my headcount because AI handles it now"?

Think about what a CISO's leverage is within a company. It's partly about the team they lead. If your headcount drops from 50 to 5, your influence in the executive suite drops with it. No C-level wants that.

And this isn't just about office politics. It's about legal obligation. Which brings us to what we think is the strongest argument in this entire article.

The Compliance Reality Check

This is the argument we think most people are missing.

On one side, fear that AI replaces SOC analysts. On the other side, a reality check. Most security programs are compliance-driven. Many orgs invest in a SOC to pass an audit, not because they love detection engineering.

So what do the compliance frameworks actually require?

We looked at SOC 2, PCI DSS, HIPAA, ISO 27001, NIS2, DORA. Here's what we found:

None of them say humans must do the triage. None of them require a human staring at alerts 24/7. They want continuous monitoring, detection capability, incident response, and evidence. They don't care if the entity doing the work is carbon-based or silicon-based.

Where humans ARE explicitly required: breach notification decisions, risk ownership, audit attestation, governance accountability. NIS2 and DORA put personal liability on executives for security failures.

DORA is the most prescriptive framework out there. Even DORA does not mandate human staffing models. It mandates outcomes.

So the question is not "will AI take SOC jobs." The question is: who signs off that the AI is doing a good job?

That person needs to exist. They need to understand what the automation does. They need to own the risk when it breaks.

When Filip posted this analysis on LinkedIn, Anton Chuvakin called it "quote of the day." And he's right, it IS a big deal. The frameworks already support AI-driven security operations. The real gap isn't technology or regulation. It's the accountability layer between the two.

So What Roles Actually Emerge From This?

The LinkedIn discussion got interesting when Rob Fry jumped in. His point: a lot of people talk about AI as though the vendor ships it, blesses it, and somehow keeps it tuned forever. That's not realistic.

Customer environments, data, workflows, risk tolerance, and operational weirdness are too specific. Vendors can provide the engine, but customers own the care and feeding. Which means you need people who can design, run, validate, and govern AI-driven security systems.

Rob described what he sees as likely new (or evolved) roles:

• Architects to design how AI fits into the control plane

• Operators to monitor, tune, and maintain it in production

• Governance folks to own risk, evidence, and accountability when it fails

• Hybrid roles that sit at the seams between security, engineering, operations, and the business

This lines up with what SACR's recent research on AI SOC and MDR shows. The market is splitting between orgs that can run AI platforms in-house (large enterprises augmenting internal teams) and those that outsource to AI-native MDR providers. Both paths need people. Different people than before, but people.

Will the SOC Analyst Become a QA Role?

Will the SOC analyst role mainly be about QA, just putting the stamp that says "looks good"?

We think partially yes. Someone has to validate the AI's conclusions. But calling it QA undersells what's actually needed. It's more like the person who signs off on the autopilot before the plane takes off. You need to understand the system deeply enough to know when it's working and when it's about to fly into a mountain. The skill set changes. The responsibility doesn't shrink.

And remember how long it took governments to even define what a "data breach" means under GDPR? Now imagine them rewriting accountability laws for fully autonomous security operations. We're not even close to that conversation in most jurisdictions. The current trend is actually toward MORE regulations that require MORE people in different regions, not fewer.

What If We're Wrong?

Fair question. Let's steelman the scary scenarios.

What if the arms race gets solved? Say we get several dozen AI SOC solutions that reach a maturity level where they integrate automatically, train themselves, and run continuous simulations that make any new attack detectable before it becomes real.

This is still sci-fi. It ignores the premise that human innovation and creativity haven't been replaced. The integration work alone is something we can't yet imagine at that scale. And the energy costs would be significant.

What if organizations stop resisting? This one we think is actually the most likely to happen eventually. The market will chip away at resistance once the technology matures. But in such a scenario, it's not just security teams that shrink. Every vertical in an enterprise would be affected, leading to smaller companies across the board.

And that creates a Ford-type dilemma: if nobody is hiring, who buys the products we're making? The "we are building AI for AI" story doesn't solve this economic question.

What if regulations change? Removing rules and regulations requiring human accountability in cybersecurity would be one part of a much larger systemic shift. If you want a historical parallel, look at Khrushchev trying to break down Soviet bureaucracy. Or Gorbachev, who arguably succeeded. Dark humor version: the operation was a success, the patient is dead. The USSR disappeared. Point being, forcefully simplifying complex regulatory systems tends to have consequences way beyond what you planned for.

And remember how long it took governments to define "data breach" under GDPR? If governments become efficient and rational enough to rewrite accountability laws across the board, the entire world would look fundamentally different. The chance of this happening in our industry alone would be a strange mathematical anomaly.

The New Shape of Things

If you've been following the secops-unpacked blog, you know we keep coming back to this: the Tier 1 SOC analyst role as we knew it is disappearing. And it should. It had the worst retention rates, the highest burnout, and it was never a real career destination. It was always a stepping stone.

But the roles replacing it are more interesting. Detection engineering. Security automation. AI validation and governance. These aren't downgrades. They're upgrades.

The three of us all started as SOC analysts. None of us do that job today. We all evolved because the industry evolved. The difference now is that the pace of that evolution is faster. But the pattern is the same: old roles get automated, new roles get created to manage and improve that automation.

Closing Thoughts

The world is changing fast. None of us know what the future holds.

But we do know this: scenarios where our industry faces massive unemployment exist. We just think they're highly unlikely within the next five to ten years.

If we reach a point where cybersecurity jobs are irrelevant in a decade, it means we've had such a massive shift in how society works that job security will be the least of our problems.

For anyone starting their career right now and wondering where to head: the compliance frameworks aren't going away. The accountability layer between AI and business outcomes isn't going away. The arms race between offense and defense isn't going away. Those are your career anchors.

The real question isn't whether AI will take your SOC job. It's whether you'll be the person who signs off that the AI is doing its job right. Position yourself for that, and you'll be fine.

What's your take? Where do you see the roles that AI can't replace in cybersecurity? What arguments do you have to say we won't be unemployed in five years? Drop a comment, we want to hear it.

Check out our SecOps Market Landscape tracker and evaluation frameworks

SecOps Market Landscape

Alright, here it is. My yearly SecOps trends and predictions report.

Yes, I know it is March. The report was written in January. Life happened. If you want timely content, subscribe to a news outlet. If you want content that ages well, stick around.

Every year I try to make sense of where this industry actually is versus where vendors say it is. Last year I called out the AI copilot hype before everyone got tired of chatbots that summarize things but do nothing. This year I have good news and bad news.

Good news: AI finally started doing real work in 2025. The "AI Analyst" is no longer a slide deck fantasy.

Bad news: We automated the easy part. And somehow created more work for humans in the process.

Let me explain.

Where we stand

MTTD improved dramatically. We got really good at finding things fast. MTTR? Still flat. We are still terrible at actually fixing things.

But here is the part that made me laugh (in a sad way): a new bottleneck emerged. Mean Time to Decision.

Before AI, a SOC processed 200 alerts daily and made maybe 50 meaningful decisions. Now AI surfaces 2,000 alerts, auto-closes 1,700, and escalates 300 requiring human judgment.

We tripled the decision load.

Congratulations. We made the SOC more efficient at generating work for humans.

Why Everyone Started in the Middle

The industry went straight for triage and investigation. Makes sense. It was the easy target. Analytically complex but operationally simple. No write access required. No change management tickets. No risk of breaking production. Just read data, make a verdict, move on.

Detection engineering? Buried in log pipelines, data normalization nightmares, and the eternal fight between coverage and alert volume.

Response? Blocked by API limitations, tribal knowledge nobody documented, and organizations that would rather accept breach risk than give AI systems write access to anything important.

So vendors went for the middle. Quick wins. Happy customers. Logos on the website.

2026 will test whether AI can shift left into detection, shift right into response, and actually reduce the human decision burden rather than just reorganize it.

The SUDA Loop

You know the OODA loop. Observe, Orient, Decide, Act. Military strategy stuff that consultants love to reference.

Security operations needs its own version: See-Understand-Decide-Act (SUDA).

Most 2025 solutions handled one or two stages. See and Understand. Or Understand and Decide. Rarely the full loop.

The platforms that win 2026 will close the entire loop. Point solutions that only serve one stage will get absorbed or left behind.

AI SOC Is Not the Platform

The industry fixated on AI SOC throughout 2025. Understandable. Alert fatigue is painful and visible. Easy to demo. Easy to measure.

But here is my take: AI SOC is one solution. It is not the platform.

The real opportunity is infrastructure that provides building blocks: agentic workflows, deterministic workflows, case management, analyst copilot, integration layer. Combine them to build any security solution your program needs.

Agentic AI SOC. The use case everyone talks about.

Agentic IAM/PAM. Access requests, privilege escalation, orphaned accounts. Identity workflows are still embarrassingly manual in 2026. Let that sink in.

Cloud Security & Vulnerability Management. Findings pile up faster than humans can prioritize. Most sit in dashboards aging like fine wine that nobody drinks.

GRC Automation. Evidence collection, control monitoring, audit prep. The work nobody wants to do, done by systems that do not complain.

Detection Engineering. Threat intel in, detection rules out, coverage gaps identified, feedback loops closed. The dream we have been chasing for years.

Threat Hunting. Continuous hunts based on intelligence and baselines. Not sporadic efforts when someone has time between incidents.

Same platform. Different solutions. Built once, deployed many times.

The Honest Take

Most 2025 AI SOC investment automated the easy part. Organizations declared victory after deploying AI triage, then discovered faster alert closure but the same remediation backlog. Often more decisions queued for human review than before.

Real maturity requires fixing detections, closing the response gap, and making feedback loops actually work. Not just closing alerts faster.

Get the Full Report

This post is the highlight reel. The full report goes deeper:

• The SecOps AI Shift Map framework for evaluating where AI actually operates

• 2025 landscape analysis across detection, triage, XDR, and automation layers

• The Workflow Gravity thesis and why it matters more than Data Gravity

• Platform building blocks and how to construct solutions beyond AI SOC

• What this means for security teams, roles, and build-vs-buy decisions

[Download: The SecOps Decision Gap - 2025 Trends & 2026 Outlook →]

Check out our SecOps Market Landscape tracker and evaluation frameworks

SecOps Market Landscape

1. Introduction

The AI SOC market is growing fast and there are products on it that are doing serious work. Some of them have strong integration capabilities, solid reasoning engines, and response actions that actually execute in production. The market has come a long way in four years.

But there is a problem with how we evaluate these products.

When every vendor says "AI-powered response," that phrase covers everything from a fully autonomous isolation workflow to a chatbot that suggests you maybe think about resetting a password. Both get the same label in the marketing material. Both show up in the same analyst reports. And when a security team sits down to compare three products, they have no standardized way to measure the gap between "our AI handles response" and what that actually means in operational terms.

Some products are close to real autonomy in specific domains. Some are strong in analysis but thin on execution. Some have broad coverage but almost nothing runs without human approval. These are all valid positions on a maturity spectrum. The problem is that there is no shared framework to place them on that spectrum consistently.

So we built one.

We call it ARMM. And yes, the name is intentional.

A decade ago, the SOAR generation solved half the problem. We built the arms. Playbooks, integrations, automated response workflows. The execution layer was there. What was missing was the brain. Every decision tree was hand-coded. Every branching logic was written by an engineer who had to anticipate every possible scenario. The arms moved, but only along rails that humans laid down manually. When the scenario deviated from the playbook, the arm froze.

Now the AI SOC generation has solved the other half. We built the brain. LLMs reason across alerts, correlate context, analyze logs, and make judgment calls that no static playbook could replicate. But somewhere along the way, a lot of products forgot to attach the arms. The reasoning is strong. The analysis is sharp. And then it hands you a summary and says "here is what you should probably do." The brain thinks. The arm does not move.

ARMM evaluates both. The reasoning quality, the decision-making maturity, the trust you can place in the AI's judgment. And the response capability, the execution depth, the ability to actually take action without three humans supervising. It weighs the arm heavier because that is where the industry gap is widest right now. But it does not ignore the brain, because an arm without a brain is just a SOAR playbook and we already know how that story ended.

ARMM is a structured scoring system for evaluating what an AI SOC solution can actually do in the response layer. It covers 80+ response capabilities across six domains: Identity, Network, Endpoint, Cloud, SaaS, and General Options. And it provides a common language so that when someone says "we handle response," there is a way to ask: at what level, across how many actions, and with what degree of autonomy?

The CyberSec Automation Blog has published over a dozen articles and podcast episodes covering what makes a good automation program succeed, how to evaluate tools, and how to structure decision-making around security automation purchases. We have built tool comparison lists, evaluation checklists, and decision frameworks. ARMM is the next step in that work.

2. Why Another Framework

Most existing evaluation methods for AI SOC solutions are either vendor-produced (and therefore biased toward their own capabilities) or too generic to capture the specific nuances of AI-driven response. Analyst reports compare products at a feature-list level without measuring automation depth. Vendor demos show best-case scenarios without exposing the operational friction underneath.

Our focus is narrow and deliberate: response capabilities. Most AI SOC solutions already deliver strong reporting and analysis features. They can summarize alerts, correlate indicators, and reduce false negatives in a mature environment (we emphasize mature because these solutions need access to quality logs and, in more advanced implementations, to organizational documentation and environment-specific context). Where the industry needs structured evaluation is in the response layer: the actions an AI SOC solution can take, how autonomously it can take them, and under what conditions.

We acknowledge that some of the capabilities listed in this framework may seem aspirational at this stage. That is by design. The framework is intended to serve both as a current-state evaluation tool and as a forward-looking roadmap.

We are not scoring specific vendors. The goal is to establish a shared methodology that allows security teams to answer questions such as:

• Which solution provides more relevant response capabilities for my environment?

• Which solution operates at a higher level of autonomy for the actions that matter to my program?

• Which solution can help me reduce my alert backlog without requiring additional headcount?

For product managers working on AI SOC products, the framework serves as a competitive analysis baseline:

• Where is my competition positioned, and what capabilities are driving their wins?

• What high-value capabilities are underserved across the market?

• Am I investing engineering resources in features that security practitioners actually prioritize?

Because this is a fast-moving space, we are starting at version 0.1. This is a living document. Version 1.0 will be designated when the framework reaches a level of stability and community validation that warrants it.

3. Scoring Methodology

ARMM supports two distinct approaches to scoring, each designed for a different operational question.

Evaluator Mode is the straightforward path. You score each capability on the 0-1-2 scale described above (with the 1C, 1G, 1A sub-levels) and the framework calculates your coverage rate, automation depth, and per-plane breakdown. The tier placements come from ARMM's reference tables. You do not need to factor in your organizational context. This mode answers one question: given two or more AI SOC products, which one covers more of what I need and at what automation level? It is built for procurement teams, SOC managers running vendor evaluations, and anyone who needs a side-by-side comparison without spending weeks on it.

Builder Mode adds a second scoring layer on top. Instead of relying on fixed reference tiers, you score each action across three axes: Trust (how much confidence does your implementation warrant), Complexity (how hard is it for your specific team to build and maintain), and Impact (what is the blast radius if something goes wrong). The action score becomes T + C + I, and the tier placement shifts based on your organizational reality. The same action that scores Entry for a mature team with established automation pipelines might score Explorer for a team that is deploying its first AI SOC integration. This mode answers a different question: given my team, my environment, and my risk tolerance, where should I invest engineering effort to move up the maturity ladder? It is built for product managers, engineering leads, and internal SOC teams running their own automation programs.

Both modes evaluate the same six planes and the same 80+ response capabilities. Both produce per-plane breakdowns and a composite maturity label. The difference is whether you want a product-level comparison (Evaluator) or an environment-aware implementation roadmap (Builder). The public ARMM app at armm.secops-unpacked.ai supports both.

3.1 The Capability Scoring System (0-1-2)

Each response capability in the framework is scored on a three-level scale that measures the degree of automation available:

0 (Not Available): The feature does not exist in the product. There is no mechanism, manual or automated, to perform this action through the AI SOC solution.

1 (Available with Human Involvement): The feature exists but requires some form of human interaction before execution. Because human involvement can range from full collaboration to a simple approval click, this level is subdivided into three sub-categories:

• 1C (Collaborator): The solution requires continuous back-and-forth interaction with an analyst to reach a response action. The AI acts as a partner, not an autonomous agent.

• 1G (Guide): The solution generates a plan and presents options for a specific action, but it is not confident in recommending a single path. It lays out alternatives and lets the analyst choose.

• 1A (Approver): The action is essentially ready to execute. The AI has determined the correct response and prepared the action, but requires a human to click approve before it fires. This is the closest step to full automation while still keeping a human in the loop.

2 (Fully Automated): The action is performed without any human involvement. The vendor (or internal implementation) has demonstrated that the AI SOC solution can execute this action with sufficient confidence that no human review is required. At the time of writing, level 2 is exceptionally rare for most response categories. The framework includes it to establish the target state and to differentiate products that are moving in that direction from those that are not.

3.2 The Three Scoring Axes (Builder Mode)

In Builder Mode, each response action is evaluated across three dimensions:

Axis 1: Decision Fidelity and Programmatic Trust (T)

This axis measures the confidence level warranted by the AI SOC implementation. It correlates directly with implementation quality: reasoning log depth, context-aware decision-making, and guardrails against hallucination.

• T = 1 (Enrichment): AI output assists human-led investigations. The AI provides context and data but does not recommend or execute actions.

• T = 2 (Validated): AI recommends a specific action. A human confirms before execution occurs.

• T = 3 (Autonomous): AI executes without human intervention. This requires the highest level of implementation maturity and organizational trust.

Axis 2: Implementation and Maintenance Complexity (C)

This axis evaluates the technical friction in building and sustaining the automation, relative to the skills and resources of the team responsible for it. This is deliberately team-dependent. An automation rated C = 3 for a junior team may be C = 2 for a team of specialized AI engineers with established CI/CD pipelines for their playbooks.

• C = 1 (Low): Simple API calls or native integrations with minimal configuration.

• C = 2 (Medium): Multi-step orchestration across multiple systems requiring coordination and testing.

• C = 3 (High): Complex behavioral baselining, legacy system integration, or custom model tuning.

Axis 3: Operational Impact and Blast Radius (I)

This axis captures the business risk associated with the action. It is typically the most stable axis across organizations, but shifts based on asset criticality. Isolating a standard employee laptop has a different blast radius than isolating a production database server.

• I = 1 (Low): Negligible disruption. Background scans, tagging, enrichment activities.

• I = 2 (Medium): Temporary disruption. Resetting a standard user session, blocking a non-critical port.

• I = 3 (High): Significant downtime, data loss risk, or reputational damage. Production system changes, VIP account modifications, critical infrastructure alterations.

3.3 The Maturity Computation Logic

The scoring system builds from individual actions up to a full program assessment through five layers. Each layer uses a defined formula.

Layer 1: Action-Level Score (S)

For a single response action, the score is the sum of its three axis values:

The minimum possible score is 3 (T=1, C=1, I=1). The maximum is 9 (T=3, C=3, I=3).

Layer 2: Tier Mapping

The action score maps to one of four maturity tiers:

Layer 3: Domain Maturity Score (D)

The maturity score for a specific domain (e.g., Endpoint, Identity) is the arithmetic mean of all action scores within that domain:

Where n is the number of scored actions in the domain. The resulting D value maps to a tier using the same thresholds from Layer 2.

Layer 4: Program Maturity Score (P)

The overall program score is the arithmetic mean of all domain scores, with equal weighting across all six planes:

Equal plane weighting is a deliberate design choice. It prevents planes with more actions (Endpoint has 22, SaaS has 10) from dominating the evaluation. Each plane contributes exactly one-sixth of the overall score.

Layer 5: Composite Maturity Label

The composite label is not derived from the program score directly. It uses sequential gating logic:

The composite label equals the highest tier where at least four out of six planes independently meet that tier's threshold, and the qualification chain is unbroken from Explorer upward. A product cannot be labeled Advanced if it has gaps at the Explorer tier.

The four-out-of-six rule is intentionally forgiving. A product focused on cloud-native environments may legitimately deprioritize network-level response. That should not disqualify it from a meaningful composite label. But it still needs breadth across most planes to earn a higher tier.

3.4 Context-Aware Scoring: Why Environment Matters

The ARMM recognizes that the maturity level of an automated action is not a static property of the feature itself. It is an emergent property of the environment where it is applied. The three axes (T, C, I) are all subject to organizational variance, which means the same product capability produces different scores in different contexts.

Example: "Isolate Device" evaluated by three different organizations using the same AI SOC product:

The product capability is identical across all three. The scores differ because the Trust axis reflects implementation maturity, the Complexity axis reflects team capability, and the Impact axis (while stable here) can shift based on asset criticality. A vendor benchmark alone is insufficient. Builder Mode exists specifically to capture this variance.

4. Response Capability Domains

The framework organizes response capabilities into six domains. The first five (Identity, Network, Endpoint, Cloud, SaaS) cover specific technical response planes. The sixth (General Options / Usability) covers platform-level characteristics that affect the operational quality of the solution independent of any specific response action.

For the first five domains, each capability is scored using the 0-1-2 system described in Section 3.1 (Evaluator Mode) or the T+C+I system described in Section 3.3 (Builder Mode). For the General Options domain, the scoring criteria shift slightly: 0 means the feature is not available, 1 means the feature is available but limited in capability or partially implemented, and 2 means the feature is fully available, functional, and tested.

4.1 Identity Response Plane

Identity-related response actions target user accounts, service principals, groups, and access permissions. These actions are among the most commonly needed in incident response and are often the first automation candidates for SOC teams.

Builder Mode Reference Scoring (Mature AI SOC Program, Skilled Engineering Team):

4.2 Network Response Plane

Network-level response actions modify traffic flow, access control, and device connectivity. These are often high-impact actions with significant blast radius, making the Trust and Impact axes particularly important in scoring.

Builder Mode Reference Scoring:

4.3 Endpoint Response Plane

Endpoint response actions operate directly on devices and their software environment. This domain has the largest number of capabilities because endpoint response spans file operations, process management, application control, forensics, and OS-level changes.

Builder Mode Reference Scoring:

4.4 Cloud Response Plane

Cloud response actions target infrastructure resources, access controls, and storage in cloud environments. The blast radius of cloud actions can be particularly severe because a single misconfigured change can affect multiple dependent services.

Builder Mode Reference Scoring:

4.5 SaaS Response Plane

SaaS response actions focus primarily on email and productivity platforms, which are among the most common attack surfaces in enterprise environments. Actions in this domain directly affect end-user workflows and communications.

Builder Mode Reference Scoring:

4.6 General Options / Usability

This domain evaluates platform-level capabilities that are not tied to any specific response action but directly affect how useful, trustworthy, and manageable the AI SOC solution is in production. The scoring for this domain uses a modified scale: 0 means not available, 1 means available but limited, and 2 means fully available and functional.

This domain is split into two sub-categories to distinguish between operational platform features and AI-specific evaluation criteria.

Platform Operations

AI-Specific Evaluation Criteria

5. Aggregate Maturity Scoring

Evaluating each plane individually is necessary but not sufficient. Security teams making purchasing decisions and product managers tracking competitive positioning need a consolidated view that communicates the overall picture without hiding the details.

5.1 Automation Depth Score

This is the most operationally significant metric and the one that separates real autonomous solutions from products that wrapped a chatbot interface around a set of API calls.

Across all covered capabilities, calculate the distribution:

• What percentage is fully automated (level 2)?

• What percentage sits at Approver level (1A)?

• What percentage sits at Guide level (1G)?

• What percentage sits at Collaborator level (1C)?

• What percentage is not available at all (0)?

A product could have 80% of capabilities covered but only 5% fully automated. That is a fundamentally different product than one with 60% covered but 40% fully automated. The first is broad but shallow. The second is narrower but operates with real autonomy where it counts.

Full Automation Rate: The percentage of total capabilities at level 2. This is the true measure of how much an AI SOC solution can operate without human intervention.

Coverage Rate: The percentage of total capabilities at any level above 0. This measures breadth regardless of automation depth.

The relationship between these two numbers tells you everything about how the product actually operates. A high coverage rate with a low automation rate means the product is a guided workflow tool with AI branding. A moderate coverage rate with a high automation rate relative to coverage means the product is autonomous in its areas of focus but limited in scope.

5.2 Combined Scoring Readout

A complete ARMM evaluation for a product produces the following consolidated output:

Per-Plane Breakdown:

6. Reading the Model

A product can reach Expert level on a specific plane by checking all the boxes for that domain. But it would be difficult to consider an AI SOC Response solution as Expert level overall if it lacks the ability to perform foundational actions like closing alerts in a SIEM. The tier system is designed to reward both depth within a domain and breadth across domains.

The reference maturity tables provided in Section 4 use example scores from a hypothetical mature AI SOC program with a skilled engineering team. These are illustrative, not universal benchmarks. The environmental dynamics described in Section 3.5 are not optional context; they are a core part of how the framework is intended to be used.

When comparing two products, the most informative comparison is not the aggregate score. It is the per-plane breakdown combined with the Automation Depth Score. Two products at the same composite tier can have radically different operational profiles. One may cover 80% of capabilities at the Collaborator level. The other may cover 50% but with 30% at full automation. These are different products for different buyers with different operational maturity levels.

7. Limitations and Future Work

This is version 0.1. The framework has known limitations:

• The capability lists are not exhaustive. New response actions will emerge as AI SOC products mature and as attack surfaces expand.

• The three-axis scoring (T, C, I) requires subjective judgment that will vary between evaluators. We plan to develop calibration guidelines to reduce inter-evaluator variance.

• The framework does not currently weight domains differently. In practice, Identity response may be more important than Network response for a given organization. Weighted scoring is planned for a future version.

• Detection and analysis capabilities are out of scope for this version. A separate framework or an extension to ARMM may address those in the future.

• We have not included pricing, deployment time, or vendor lock-in considerations. These are important purchase factors but are outside the scope of a technical maturity model.

We are building a public web application where users can input their product's capabilities and generate ARMM scoring layers automatically, along with an exportable CSV. The application is available at: armm.secops-unpacked.ai

8. Conclusion

The AI SOC market is growing faster than the industry's ability to evaluate products on consistent terms. The ARMM framework provides a structured, repeatable methodology for measuring what an AI SOC solution can actually do in the response layer, how autonomously it can do it, and what it takes to deploy and maintain that capability in a specific operational environment.

The framework is built for two audiences: security teams evaluating products and product managers building them. For security teams, it provides a checklist and scoring system that cuts through marketing language and focuses on operational capability. For product teams, it provides a competitive analysis baseline and a prioritization framework for feature development.

SOAR gave us arms without brains. The first wave of AI SOC products gave us brains without arms. The products that will win this market are the ones that connect both. ARMM gives you a way to measure how far along that connection is, and where the gaps remain.

No current AI SOC solution will check every box. That is not the point. The point is to establish a common language and a common measurement system so that the conversation about AI SOC response capability is grounded in specifics rather than promises. Version 0.1 is the starting point. The framework will evolve as the market does.

Check out our SecOps Market Landscape tracker and evaluation frameworks

SecOps Market Landscape

The Fear of Not Doing Enough: Security's Workflow Problem

If you've been following this blog, you know I've spent a lot of time on AI transforming investigation, triage, and detection engineering. And a few months back I wrote about the single pane of glass, how it's not a product you buy but a system you build, piece by piece, like Legos.

That post was about the architecture. What tools do you need, how do they connect, where does data flow.

This post is about the layer underneath that nobody talks about. Not the tools. The work itself. Where it comes from, how it flows, and why we have zero visibility into most of it.

The Fear of Not Doing Enough

Security has this pattern I keep seeing everywhere.

New attack technique drops. A CVE trends on Twitter. Some threat intel report lands in your inbox with a fancy APT name. What happens next? Predictable.

Someone writes a generic detection rule fast so the team has "something." Gets pushed to production. Generates noise. Nobody tunes it because there's already another thing screaming for attention.

The false sense of coverage becomes more important than actual coverage.

I call this the Fear of Not Doing Enough. And honestly? It drives most of the operational pain in security teams today.

You write a detection rule. Do you have the SOP for when it fires? Do you know the full analysis path an analyst should follow? Can you estimate how that alert impacts your team's workload downstream? Do you know what "done" looks like for that alert type?

If you can't answer those, you didn't deploy a detection. You deployed a work generator with no operating manual. Multiply that across dozens of detections written under pressure and you get patchwork coverage that looks great on a dashboard but falls apart when someone has to actually operate it.

But here's the thing. Even if you fix all of that, you're still only looking at one input stream.

It's Not Just SIEM Alerts

A ACM Computing Surveys paper (Tariq et al., 2025) reviewed over 30 solutions to alert fatigue in SOCs. Thorough paper, I'll give them that. Identifies four root causes: staff shortage, high false positive rates, disconnected dashboards, and inefficient SOPs.

But every single solution assumes the work starts with a SIEM alert.

Now look, I'm not saying SIEM alerts are a small part of the work. For most teams they're probably more than half. But here's what matters: the work that doesn't come from the SIEM is often the most manual, least structured, and hardest to track.

IT escalations. Someone from the help desk pings you on Slack: "Hey, this looks weird." Access review requests from HR. Audit findings that need remediation tracking. Pen test findings that need to be assigned and fixed. Third-party risk questionnaires. Compliance asks from legal.

All real security work. And here's the thing about it: only some or none of it has a playbook or automation behind it. Most of it lives in Slack threads, email chains, and spreadsheets. It's the security work that runs entirely on copy-paste, tribal knowledge, and good intentions.

Your SIEM alerts, for all their problems, at least flow through a pipeline. They get enriched. They have some structure. Maybe even a SOAR playbook attached. The non-SIEM work? It's the Wild West.

Erik Bloch has been making this point for years.A lot of the work SOC is doing day-to-day has nothing to do with chasing advanced adversaries. It's tickets, reports, evidence collection, reconciling data across tools. The mundane operational grind that actually burns people out.

And here's the part that really gets me. Outside of very large enterprises that have 10 security sub-departments with dedicated teams for everything, the same 3-5 people triaging SIEM alerts are also pulling evidence for the auditor, handling the IT escalation, and answering the compliance questionnaire. There's no luxury of specialization. The alert queue is just one input stream among many. And the non-SIEM stuff eats time disproportionately because it's all manual.

Security Work Has No Gravity

Ross Haleliuk recently wrote a great piece about ServiceNow betting on "workflow gravity" to compete with the security platform giants. The thesis is simple. Whoever owns where work happens owns the decisions.

Data gravity pulls information into a single system of record. Your SIEM, your data lake, whatever. That part most teams have figured out. Workflow gravity is different. It pulls action into a single system of action. One place where work lands, gets triaged, gets tracked, and gets done.

Right now? Security work has no gravity. It's everywhere and nowhere.

And yeah, this connects directly to the single pane of glass conversation. In that post I talked about building your own platform, Lego-style, with assets, data layers, correlation, and response actions. But even if you build that beautiful architecture, it's still oriented around machine-generated alerts. The SIEM brain, the enrichment layer, the correlation engine. All of that assumes the input is a structured alert.

What about the IT manager who emails you about a suspicious contractor? What about the audit finding that needs 6 teams to remediate? What about the pen test report sitting in a shared drive that nobody has turned into action items yet?

That work has no architecture. It has no pipeline. It just shows up and someone deals with it however they can.

You want to know why security teams always feel understaffed? Part of it is real headcount shortage, sure. But part of it is that nobody can actually see where the time goes. When the most manual, time-consuming work lives outside of every system you've built, you can't measure it. When you can't measure it, you can't optimize it. When you can't optimize it, you just throw more people at it and hope for the best.

Process Mining Exists. Just Not for Us. Yet.

Here's something that gets me. In finance, procurement, and operations, tools like Celonis and Scribe Optimize have existed for years. They observe how work actually happens across tools and systems. They find bottlenecks. They tell you where time is wasted. They optimize based on data, not vibes and assumptions.

In security? Still very early days.

Some vendors are starting to take RPA-style approaches to There's a handful of academic papers exploring it. But it's nowhere near mainstream.

We still don't have good data on how security work actually flows end to end. Think about that.

We have terabytes of security telemetry. We can tell you exactly when a process spawned on an endpoint at 3:47am. But we can't tell you how long it takes an analyst to go from "alert fired" to "investigation complete." We can't tell you how much time the team spends on compliance requests versus actual threat work. We can't tell you which of your 200 detection rules generates the most operational overhead relative to the security value it provides.

That's wild.

Why This Is Hard

I get why the industry keeps gravitating toward the easier wins. Make investigation faster. Automate the playbook. Build a better ML model for triage. Those are well-defined problems with measurable outcomes.

Understanding where all security work happens and how it flows? That's messy. It crosses tool boundaries. It involves human behavior that doesn't fit neatly into event logs. It requires looking at the whole system, not just one piece.

This is the hardest problem to solve. And that's exactly why not many are tackling it yet.

But here's why it matters. If you don't understand the full picture of how work enters and flows through your security team, everything else you build is an optimization of a subsystem. You can make SIEM triage 10x faster, but if a third of the work comes from non-SIEM sources that are entirely manual, you just made one part of the problem better while the messiest part stays untouched.

What Would Actually Help

I don't think this needs to be one giant platform that replaces everything. But teams need a few things that barely exist today.

Workflow data. How long does each type of work actually take? Where are the handoffs? Where do things stall? What percentage of the team's time goes to which category of work? Right now most teams are guessing. And the guesses are usually wrong because the most painful work is the least visible.

Operational impact awareness. Before you deploy a new detection, onboard a new data source, or agree to a new compliance requirement, you should be able to model what that does to your team's capacity. Not after the fact when everyone's drowning. Before.

Connection between detection and process. If you have a detection but you don't have the analysis path mapped from it, you can't estimate how it impacts anything downstream. Every detection should ship with its SOP. Not as a nice-to-have. As a requirement.

The Fear Won't Go Away

The Fear of Not Doing Enough will always be there. New threats aren't going to stop coming. The pressure to have "something" for every new attack vector is real.

But the answer isn't to keep throwing generic detections at every new thing and hoping the team can absorb the blast. It's not to keep building faster investigation tools for one slice of the work while the rest drowns in Slack threads and spreadsheets.

We've been fixing the middle. Investigation is getting faster. AI triage is real. Response automation is improving. The single pane of glass architecture is getting clearer. All good progress.

Now it's time to zoom out. Understand how security work actually flows. All of it. Not just the structured, machine-generated part. Especially the messy, manual, human-generated part that eats the most time and has the least tooling.

Fix the input. Model the cost. Understand the workflow.

Stop optimizing the output of a system you've never fully mapped.

Check out our SecOps Market Landscape tracker and evaluation frameworks

SecOps Market Landscape

Hey everyone. Been quiet on here for a bit. First post of 2026.

I came across a paper that finally articulates something I've been thinking about for a while: autonomy isn't a capability score. It's a design decision"Levels of Autonomy for AI Agents". And chasing the highest level everywhere is a mistake.

Let me walk through it.

L1 (Operator): User directs and makes decisions, agent acts. Think Microsoft Copilot. The agent requires invocation to act, provides on-demand assistance, and avoids preference-based decision-making on the user's behalf.

L2 (Collaborator): User and agent collaboratively plan, delegate, and execute. Think OpenAI Operator. Users can freely modify agent work and take control at any point. Back-and-forth communication is frequent.

L3 (Consultant): Agent takes the lead but consults user for expertise and preferences. Think Gemini Deep Research. Users provide feedback and directional guidance rather than hands-on collaboration. The agent bears more of the learning curve.

L4 (Approver): Agent engages user only in risky or pre-specified scenarios. Think Devin. Users specify approval conditions upfront. The agent only stops for blockers, credentials, or consequential actions.

L5 (Observer): Agent operates with full autonomy under user monitoring. Users can watch activity logs and hit the emergency stop. That's it.

The key insight: autonomy is a design decision, not a capability metric. A capable agent can still operate at L2 if that's the right call for the task. The paper explicitly argues against treating autonomy as an inevitable consequence of increasing capability.

Agency vs. Autonomy

The paper makes an important distinction that matters for security operations.

Agency is the capacity to carry out intentional actions. It's about what tools the agent has access to and what it can do in the environment.

Autonomy is the extent to which the agent operates without user involvement. It's about when and how the agent checks in with humans.

An agent with high agency (many tools, broad permissions) can still have low autonomy (checks in frequently). An agent with low agency (limited toolset) can have high autonomy (runs independently within that scope).

This distinction matters because security teams often conflate the two. Giving an agent access to more data sources (agency) is different from letting it act without approval (autonomy). You can expand agency while constraining autonomy.

Mapping Autonomy to Security Operations

I mapped common security workflows to appropriate autonomy levels based on their risk profile and decision complexity.

Initial Triage: L4-L5

For alert triage, scale beats precision. You're dealing with volume. The goal is filtering, not final judgment.

L4 makes sense here. Let agents do the heavy lifting, have them seek approval only for edge cases or high-severity alerts. L5 is reasonable for low-fidelity alerts where false positives cost nothing. Start at L4. Keep humans reviewing outcomes for anything that escalates to investigation.

The paper notes that L4 agents are ideal for tasks with high amounts of lower-stakes decision-making. Alert triage fits this description. Automated decisions improve efficiency. Erroneous decisions on individual alerts don't impose catastrophic risks if the escalation path is intact.

Incident Response: L1-L2

On the response side of the IR cycle, L1 and L2 work well. These are established patterns with runbooks and playbooks.

Why keep autonomy low here? Response actions are consequential. Isolating a host, blocking a domain, killing a process. These actions have real impact. Speed matters, but accountability matters more.

L1 is appropriate when analysts drive the workflow and agents execute specific tasks on command. L2 works when you want the agent to propose containment plans while the analyst retains takeover capability.

The paper describes L2 control mechanisms as requiring "control transfer from agent to user, and vice versa" plus "shared representation of progress." That maps well to incident response dashboards where analysts can see what the agent is doing and intervene.

In-Depth Investigation: L3

Deep investigation is judgment-heavy work. Context matters. The analyst brings domain knowledge, institutional memory, and threat intelligence the agent doesn't have.

L3 fits this workflow. The agent leads the investigation, gathering data, correlating events, building timelines. But it consults the analyst for direction. What's the hypothesis? Which threads are worth pulling? Does this pattern match something we've seen before?

The paper notes that L3 agents require "productive and timely consultation." The agent needs to know what expertise the user brings and when to ask for it. For security investigations, this means the agent should surface findings and ask about relevance rather than drawing conclusions autonomously.

Threat Hunting: L2-L3

Hunting is exploratory by nature. You're looking for things you don't know exist yet. Hypotheses matter. Intuition matters.

Collaboration beats full automation here. L2-L3 is the range. The agent surfaces anomalies, suggests investigation paths, runs queries. The human drives the hunt itself.

The paper describes L2 as the level where "back-and-forth communication between the user and the agent is the most frequent and rich." Threat hunting benefits from this dynamic. The hunter's domain expertise combined with the agent's ability to process large datasets creates a feedback loop that pure automation can't replicate.

Detection Engineering: L2-L3

Detection engineering is systematic but consequential. Bad detections create alert fatigue. Missed detections create gaps.

L2 is the baseline. The agent assists with query building, suggests detection patterns, helps test against historical data. The engineer retains control over what gets deployed.

L3 is appropriate for mature teams with well-governed detection lifecycles. The agent drafts detections, runs validation, and consults the engineer before deployment. The key is having proper testing and review controls already in place.

The paper warns about L4 agents and "meaningless rubber stamping" from user disengagement. This risk is real for detection engineering. If engineers just approve whatever the agent proposes, detection quality will degrade.

The Double-Edged Sword

The paper repeatedly emphasizes that autonomy amplifies both benefits and risks. Higher autonomy means more scale and efficiency. It also means errors compound over multiple steps without intervention.

This maps directly to security operations. An agent that autonomously closes false positive alerts at L5 saves analyst time. An agent that autonomously closes true positive alerts at L5 creates security incidents.

The paper also raises concerns about deskilling and loss of critical thinking when automation takes over judgment tasks. Security teams should consider this. If agents handle all investigation, what happens to analyst skill development? L2 and L3 autonomy levels preserve opportunities for human engagement while still providing automation benefits.

Autonomy Certificates

The paper proposes "autonomy certificates" as a governance mechanism. A third-party body evaluates an agent's behavior and certifies the maximum autonomy level at which it can operate.

This concept has implications for security vendors. Right now, every AI SOC vendor claims some version of autonomous operation. There's no standard way to compare what that actually means.

An autonomy certificate framework would force clarity. Does your agent operate at L3 or L4? What approval mechanisms exist? Under what conditions does it escalate?

For security buyers, this creates better evaluation criteria than vague claims about AI capabilities.

Double-Layer Governance: Reasoning and Abilities

The agency vs. autonomy distinction from the paper points to a practical governance model. You need to control both what the agent can think about doing and what it can actually do.

At BlinkOps, we implement this as double-layer governance:

Layer 1: Reasoning Constraints

This layer limits what the agent can decide to do. It's autonomy governance. You define the scope of problems the agent is allowed to reason about and the types of conclusions it can reach.

For example, an agent handling alert triage might be constrained to reason only about severity classification and enrichment. It can't decide to initiate response actions, even if it has the technical capability. The reasoning boundary is set before the agent ever considers what actions to take.

This maps to the paper's definition of autonomy as "the extent to which an AI agent is designed to operate without user involvement." By constraining reasoning scope, you limit how far the agent goes before involving a human.

Layer 2: Ability Constraints

This layer limits what the agent can execute. It's agency governance. Even if the agent reasons its way to a valid conclusion, it can only act through explicitly permitted capabilities.

This is your tool allowlist. The agent might determine that isolating a host is the right response, but if host isolation isn't in its permitted action set, it can't execute. It has to escalate.

This maps to the paper's definition of agency as "the capacity to carry out intentional actions." By constraining the toolset, you bound the blast radius of any autonomous decision.

Why Both Layers Matter

Single-layer governance creates gaps.

If you only constrain abilities (Layer 2), the agent can still reason about actions outside its scope and make recommendations that push humans toward decisions the agent shouldn't influence. An agent without response permissions might still conclude "this host should be isolated immediately" and create pressure for hasty action.

If you only constrain reasoning (Layer 1), the agent might find edge cases where its reasoning scope overlaps with dangerous capabilities. A triage agent reasoning about "enrichment" might decide that querying a production database for context falls within scope.

Double-layer governance closes both gaps. The reasoning layer defines intent boundaries. The ability layer enforces execution boundaries. An action only happens if it passes both checks.

Practical Implementation

For each workflow, define:

1. Reasoning scope: What questions can the agent answer? What conclusions can it reach? What types of decisions are out of bounds?

2. Action permissions: What tools and integrations can the agent invoke? What parameters can it set? What requires human approval?

3. Escalation triggers: When reasoning hits scope boundaries, where does it go? When actions require approval, who approves?

This gives you granular control without blocking automation entirely. An L4 agent can still operate autonomously within its defined scope. But that scope is explicitly bounded at both the reasoning and execution layers.

The paper's framework helps here. L4 requires "customizable conditions for seeking approval." Double-layer governance operationalizes this. The conditions are defined by reasoning scope violations (Layer 1) and action permission requirements (Layer 2).

What This Means for AI SOC Design

If you're building or buying AI-powered security tooling, ask different questions:

What autonomy level does this workflow need? Not "how autonomous is this agent?" Match the autonomy to the task risk profile.

What are the must-have controls? Each autonomy level has required control mechanisms. L4 requires approval elicitation for consequential actions and customizable conditions. L2 requires control transfer mechanisms and shared progress visibility. Verify these exist.

Where are the approval gates? Every workflow should have defined checkpoints. Know what triggers human involvement.

What's the fallback? When the agent hits a failure state, what happens? The paper notes that L4 and L5 agents should iterate on solutions or modify approaches when blocked. How does your agent handle this?

Who's accountable? Higher autonomy means harder accountability tracing. The paper cites research showing it's simultaneously more important and more difficult to anticipate harms from autonomous AI. Design governance around this reality.

Closing Thoughts

Chasing L5 everywhere is a design mistake, not a strategy.

The vendors pushing "fully autonomous SOC" are selling a destination most teams shouldn't want to reach. The right autonomy level varies by task, by maturity, by risk tolerance.

The paper's framework gives us a shared vocabulary for these discussions. Use it.

Reference: Feng, K.J.K., McDonald, D.W., & Zhang, A.X. (2025). Levels of Autonomy for AI Agents. University of Washington. arXiv:2506.12469

Check out our SecOps Market Landscape tracker and evaluation frameworks

SecOps Market Landscape