Executive Summary

• Evaluating a security agent begins with defining the work it should perform and the evidence required to consider that work complete and accurate. Domain experts establish reference answers and grading criteria. 
• A successful approach must combine evaluations of complete runs alongside tests of individual competencies to both demonstrate whether a system is useful and help engineers identify what exactly to improve.
• Every material change to a model, prompt, tool, or workflow needs a measured comparison with the previous system. Operational failures supply new test cases, and fresh cases test whether improvements generalize.

In Part 1 of LLMs in the SOC, SentinelLABS examined the gap between cybersecurity benchmark scores and operational usefulness. In the months that have followed, we’ve observed those same design gaps in the flood of agentic systems hitting the market. This fail-fast development trend treats evaluation as a process that emerges after the proof-of-concept. In other words, teams build the system, demonstrate that it can work, and defer the harder task of establishing how reliably it performs and where it fails for a later date. 

This makes sense in the new world of rapid prototyping and AI experimentation. However, these systems are often brittle or only work until something in the environment changes or fails to generalize. This raises the urgent question, "How do I fix or improve this one behavior?" inside of a monolithic, non-deterministic pipeline.

For a system that will support security decisions, we believe the most important design choice is to make its performance measurable from the start. That means building the agent so that engineers can identify where an investigation goes wrong, test a correction, and verify that the change improves the complete workflow.

Consider a grounded example: an investigation agent that reviews an alert, queries the SIEM, and returns a verdict that matches an experienced analyst's. That looks like a success. On closer inspection, however, the query used the wrong device identifier and retrieved unrelated events. The verdict may be correct, but the records gathered during the investigation do not support the decision.

How would the team building that agent discover the mistake? And after fixing it, how would they know that the next model or harness update had not brought it back?

Start With the Work the Agent Should Complete

When building agents, attention often goes to the visible machinery: which model, which prompts, which tools, and so on. Those matter, but they're also some of the easier parts to modify. The part that's hardest to retrofit is the ability to answer two questions: is the system effective? And if not, what exactly do I change to make it better?

The industry has converged on this from several directions. Frontier lab engineering guidance observes that teams get surprisingly far on manual testing and intuition, but once an agent is in production, building without evaluations breaks down. Users report that the agent "feels worse" after a change, engineering scrambles to find out whether the claim has any validity, and it becomes difficult to tell real regressions from noise or to test changes against realistic scenarios before shipping [2]. For some the answer is what they label eval-driven development, scoped tests at every stage, before and during implementation, the way test-driven development writes tests before the code exists [3].

What that guidance usually leaves implicit is that evaluation discipline is an architectural decision. If you want to measure components individually with rigor, the agent has to be built out of individually exercisable competencies.

Before we choose a model or write a prompt, we need to define the work we expect the agent to do. For an investigation agent, that means reviewing an alert, gathering the evidence needed to understand what happened, and reaching a conclusion that the evidence supports. When it cannot resolve the case, it should give an analyst a clear account of what it found and what still needs investigation.

On Evaluations

"The first principle is that you must not fool yourself, and you are the easiest person to fool." – Richard Feynman

An evaluation is a way to measure a model's performance against a given scenario. More formally, every evaluation decomposes into a task (the point of indecision the system must reason through), a scenario (the environment wrapped around that task), and rubrics (how completion gets judged). This may seem simple on the surface, but LLM evaluations are an emerging discipline that every industry is struggling with. As the Pydantic team puts it, "Anyone who claims to know exactly how your evals should be defined can safely be ignored" [4]. A good eval set asks "what will a team use this for?" and ensures that the system gives them sufficient answers, and that it does so consistently.

End-to-End Evaluation: Does the System Perform?

An end-to-end evaluation runs the whole system on a realistic workload with grades as the final outcome. For an investigation agent, the natural frame is the human baseline it augments. A SOC analyst starts the day with a queue. Over a shift they work some number of alerts, dispositioning them into buckets (benign, suspicious, escalate, respond), and they end the day with a measurable amount of work left over. One metric that matches is burndown: of everything the system started with, how much still needs human attention? Alongside it sit the metrics that keep this burndown honest, such as disposition accuracy versus expert ground truth, time-to-verdict, stability, consistency, and cost per alert.

End-to-end metrics like these are often seen as the only metrics that matter. They tell a compelling story, they demonstrate that a system is worth building, and they're exactly the narrative that users want to hear. They also catch emergent failures such as compounding errors, context loss, and mis-sequenced steps that only show up when every component runs together against realistic inputs [2,6].

However, end-to-end metrics can aggregate away the information an engineer needs. When the overall performance is bad, or when it degrades after a change, it doesn’t provide any context about what went wrong. A regression in burndown could indicate one of many problems, meanwhile, all we can measure here is that suddenly we aren't performing. In a monolithic system, the only answer is to start reading traces and hunting for sources of failure.

Competency Evaluation: What Exactly Do We Fix?

To tackle that question we use the competency evaluation (elsewhere called component-level [6]). A competency evaluation isolates one skill and tests it directly, with its own specific inputs, definition of success, and a wide variety of different cases. These evaluations are helpful because they tell you whether a single part of a larger system is functional and capable across many scenarios. The inverse is also true, which is to say that it tells you which exact skills are underperforming.

End-to-End and Competency Evaluations: What Each Reveals

A mature program runs both continuously, and many teams add additional layers. For example, trajectory or trace evaluation, which grades the path the agent took rather than only the endpoints [2,6]. These evaluations are also critical, but this article specifically serves to spotlight the two defined above.

A Real-World Example

Take our Purple AI® Agentic Investigation agent as an example. When invoked, the system runs the investigation in the same way an analyst would. It pulls the surrounding evidence for an alert, enriches that evidence with additional data, forms and submits queries against the organizations telemetry, reasons over what comes back, and lands a verdict with the findings to support it. The tempting way to grade that answer is obvious, does the verdict match what an expert would have decided?

Very tempting, and also very dangerous. Suppose an agent agrees with the expert 95% of the time. That sounds strong, but the agent could just be pattern-matching on alert characteristics instead of actually retrieving evidence and building intermediate conclusions. A correct verdict reached for the wrong reasons is a failure, and only having a verdict-level score doesn’t tell us how we arrived at a conclusion. This is the same blind spot SentinelLABS identified in Part 1 - the unit of evaluation was more than often the answer to a question, as opposed to the path taken to reach it.

Decomposing the Problem

So, how do you evaluate competencies in this example? Start from the job you're trying to solve and decompose. On inspection, investigation hinges on many competencies. The agent has to form hypotheses worth testing given alert data and state from previous steps. It has to translate those hypotheses into correct and efficient queries against each data source, and it has to understand the schema of that source and what the fields it's filtering on actually mean. It has to interpret result sets, pick the next pivot, and know when the evidence is sufficient to stop. 

Each of these is a distinct competency that may be evaluated on its own. 

An (extremely abbreviated) tree:

Investigation 

├─ Hypothesis generation

├─ SIEM query construction (per-source eval suites)

├─ Investigation efficiency (tool-call budget)

├─ Adherence to organizational policy

├─ Confidence & evidence sufficiency judgment

└─ etc.

Now our evaluations can become concrete. "SIEM query construction" means many cases, each specifying an investigative intent, a target data source, and ground truth for what a correct query returns, graded automatically by running the query against representative data and comparing results, and backed by experts who wrote their own queries for the same asks.

For example, consider the device-ID failure we mentioned above. A competency case could give the agent an alert with a device ID and ask it to retrieve the related authentication events from a representative SIEM dataset. The case already defines which events a correct query should return.

If the agent queries the wrong identifier, it fails the case. The query may run successfully and return plausible data, but it does not return the expected events. This turns a failure that was previously hidden inside a successful investigation into something the team can measure directly.

Once the team fixes the problem, the case stays in the evaluation suite to ensure future changes to the system are consistently measured.

Even just a few dozen well-chosen evaluations per capability, drawn from real failures, can provide a strong initial signal. Each of these competencies gets its own suite since we know they are areas where models may struggle.

Where to Invest Evaluation Effort First

Not every node in the tree deserves equal attention. When setting up your own evaluations we suggest asking two questions:

Is this actually hard for models? Some competencies are commodities today. Summarization is the canonical example. Modern LLMs summarize nearly anything competently and the difference between a good and a great summary rarely moves outcomes. Contrast that with SIEM query construction, a real technical hurdle, with objective failure (the query is wrong, slow, or returns the wrong rows), high downstream impact (a bad query corrupts the whole investigation), and known model weakness on dialect and schema-specific syntax. An area that exhibits similar challenges is where your evaluation cases should go.

Can you define a defensible standard? Some competencies rely on direct checks against known facts, while others require expert judgement about whether an agent’s decision was justified by the evidence available. Evaluation effort pays off where success is decidable. Query construction has crisp ground truth (run it, compare results). Evidence-sufficiency judgment is softer, but experts mostly agree on clear cases, so a rubric plus expert-labeled examples works. Where even experts can't agree what "good" means, an eval suite just encodes noise. Before evaluating, you must first have a clear definition of success [10,11].

A useful heuristic is to simply combine the two and prioritize competencies by impact of failure × probability of model failure × decidability of ground truth. Wherever possible, look to grade binary pass/fail per case instead of arbitrary quality scores as binary judgments are easier for experts to make consistently, easier to trend, and harder to argue with [10]. If your system is already in the wild, you can still build cases - but we recommend you prioritize observed failures over successes [2,3].

The Value of Domain Expertise

Every competency evaluation needs to answer the question "what does good look like?" The person who can say whether a generated query succeeded (not just executed, but "all things considered, this is equal to or better than what I would have written in my day job") is the person who writes those queries well.

This is why many companies are spending heavily on contracted expert annotators. They're buying, at market rates and arm's length, the domain judgment that their evaluations require. In reality, most AI engineers are not domain experts, and many teams building agents don't have these resources in abundance. This is where being a company of cyber professionals is our advantage: for nearly any skill a security agent needs, someone here already performs it at an expert level daily. That expertise is the standard every one of our evaluations is built against.

When you identify a competency, identify its experts, understand how the work actually functions, and co-author cases with them. Evaluations built this way are true to real life, benefiting generalization when it reaches the field as it was tested against reality rather than the builder's imagination of what the field might look like.

Prove It Before Production

A well-built competency tree carries a risk because it makes the evaluation effort feel finished. However, evaluation is a constant game of discovering scenarios, hill climbing performance, and monitoring for regressions. Inside SentinelOne®, once an agent has been well-evaluated, it earns the chance to prove itself alongside real security work. 

At this stage, we actively execute solutions in parallel with working analysts in our internal SOC and Wayfinder Managed Services teams. When runs are flagged for review, an expert re-investigates the alert from scratch and establishes ground truth before reading the agent's verdict, so the reference standard can never be anchored by the machine it's meant to judge. 

Every interesting divergence becomes a new case, and every fix then has to beat the case that motivated it while not regressing against the larger suite. Only after our internal teams have thoroughly vetted an agentic offering in production does it reach anyone outside SentinelOne.

Underneath all of this effort sits one standard: in security response, unmeasurable capability is unshippable capability. An agent that cannot demonstrate exactly where it is weak cannot be trusted where it is strong. Autonomy in a SOC must be earned competency by competency, evaluation by evaluation, and finally in production, in front of the people who do the job.

References

[1] G. Bernadett-Shapiro and E. Garcia Lazo. LLMs in the SOC (Part 1) | Why Benchmarks Fail Security Operations Teams. SentinelLABS, January 2026.

[2] Anthropic. Demystifying Evals for AI Agents. Anthropic Engineering, January 2026.

[3] OpenAI. Evaluation Best Practices. OpenAI API Documentation.

[4] Pydantic. Pydantic Evals. Pydantic AI Documentation.

[5] OpenAI. Advancing the Price-Performance Frontier with GPT-5.6. OpenAI, July 2026.

[6] Confident AI. LLM Agent Evaluation Metrics: Tool Calling, Task Completion, Reasoning, and Trace-Based Evals. Confident AI, 2026.

[7] R. Aleithan et al. SWE-Bench+: Enhanced Coding Benchmark for LLMs. arXiv:2410.06992, 2024.

[8] Anthropic. Writing Effective Tools for AI Agents — Using AI Agents. Anthropic Engineering, 2025.

[9] Anthropic. Building Effective AI Agents. Anthropic Research, December 2024.

[10] H. Husain. Your AI Product Needs Evals. hamel.dev, 2024.

[11] A. Szymanski et al. Limitations of the LLM-as-a-Judge Approach for Evaluating LLM Outputs in Expert Knowledge Tasks. Proceedings of the 30th International Conference on Intelligent User Interfaces (IUI), 2025.

Third Party Disclaimer:

All third-party product names, logos, and brands mentioned in this publication are the property of their respective owners and are for identification purposes only. Use of these names, logos, and brands does not imply affiliation, endorsement, sponsorship, or association with the third-party.

On August 14, 2026, Alibaba released Qwen3.8-27B, a 27-billion-parameter open-weight model small enough to run on a laptop. Seventy-two hours later, the first fine-tune had already stripped its safety guardrails. On August 20, a researcher published an "abliterated" version, claiming, by his own testing, a 0% refusal rate across 842 prompts that safety teams are specifically trained to catch. A day after that, a demo of that stripped model running on a Mac, answering exactly the kind of request it was supposed to refuse, with no jailbreak engineering required, was making its way around the industry.

Near-frontier model capability is showing up on hardware anyone can buy within days of release. Months of a lab's safety work can be stripped out by someone else's side project just as fast. If you already have SOC data, and you already have engineers who can write a prompt, the AI SOC you'd otherwise buy starts to look like a sprint or two, not a purchase decision. That's why "let's just build our own AI SOC" sounds like a weekend project right now. 

The aspect often not considered is that the moment you stand up your own model, write your own routing logic, and point it at your own alert queue, you've accepted a job. Congratulations, you're now a security vendor. Any company that has ever sold you a SOC tool has accepted a long list of obligations on your behalf: supporting it, patching it, improving it, and acting on feature requests, the kind of list you rarely had to think about. Build your own version, and that list transfers to you, in full, starting today.

What follows is the fine print of what you have now accepted: the costs, the ceiling on how independent "truly DIY" really makes you, and the oversight that never gets to be optional. The goal here is not to settle a build-versus-buy debate. It's to contrast what any point on that spectrum actually puts on your plate operationally once AI enters the SOC, including how much of it remains human work, no matter who owns it.

Buy, Build, or Somewhere In Between

Buying a platform or building your own looks like a binary choice. It's actually a spectrum, and most organizations are already standing on a point along it without having picked that point on purpose. The hidden costs don't apply evenly across that spectrum. They change shape depending on where you land, and on how much sustained engineering investment security already gets inside your organization. A handful of teams have that investment. Most don't.

• Buy. A platform investigates, routes, and maintains itself on your behalf. You take on very little of what follows in this piece in exchange for working within the boundaries the platform sets. Which data sources it connects to, which models it has already vetted, and how quickly its roadmap moves relative to yours.

• Build-with. You orchestrate agents, MCP servers, or tools on top of infrastructure someone else still runs, patches, and keeps online. You gain real control over the workflow itself: how alerts are routed, what counts as a good verdict for your team, and what a playbook actually does, step by step. That maintenance burden simply relocates, moving from "who runs the model" to "who wires these pieces together correctly, and keeps them wired correctly as every piece changes underneath you." Layering AI on top doesn't fix a slow rate-determining step underneath it. Federated queries against a legacy SIEM architecture are still slow. A disconnect between identity and endpoint still delays remediation. And an agent that can't pull fresh forensic telemetry off an endpoint in real time is reasoning about stale information, no matter how good the model is.

• Truly DIY. Your own model, your own routing, your own infrastructure, built from the ground up. That last part reaches past the application layer, down to the hardware itself, the operating system running it, and every failure or patch cycle that comes with owning both. All of it is now yours to run and defend. Every decision is yours to make, which also means every consequence of that decision is yours to own, on a timeline nobody else is tracking for you.

From the ground up has a real ceiling, and it moves. The frontier you build against today won't be the frontier you're maintaining against once your stack is stable. Closing that gap every time a new model, a new attack pattern, or a new class of threat shows up is a standing commitment with no finish line, and that's a different kind of undertaking than the one-time build most teams picture when they say "truly DIY." Almost nobody actually gets to stop building. The real question is whether you're willing to keep chasing truly DIY indefinitely, on a timeline that resets every time the frontier does.

The far end always sounds smaller than it actually is. "Why not just put it in Amazon S3 with scheduled queries?" is a fair question, and also the first domino. The moment a scheduled query against a growing log store becomes part of your detection or investigation path, you've taken on retention design, query performance at scale, freshness lag, and a cron job as a piece of your incident response infrastructure. Nobody budgeted for that as a security commitment because it didn't arrive labeled as one.

The Job Description You Just Signed

These costs show up as dollars sometimes, but just as often, they show up as time and as coverage gaps that nobody notices until they matter. They don't all carry the same weight. Most of what follows is friction, real friction, the kind that slows a roadmap and eats afternoons, nights, and weekends. One of them is the one that ends up on a postmortem with your name on it.

The friction you'll live with day to day:

Model Choice and Routing

Picking wrong here doesn't announce itself. You find out weeks or months later, usually during an incident, that the model missed something a different one would have caught. Every model swap, frontier to open-weight to whatever ships next, adds a new maintenance task to every workflow already built on top of the old one, and catching a bad pick early means running your own ongoing benchmarking practice against your own data. 
Done well, this pays off. RouteLLM (ICLR 2025) showed that a router can cut costs by more than half while retaining roughly 95 percent of a flagship model's quality on standard benchmarks by sending easy queries to a cheaper model and reserving the expensive one for queries that actually need it. Routing has become its own layer of infrastructure. Whether you build that layer yourself, buy it, or inherit it through your vendor, someone still has to maintain it as the frontier moves.

Infrastructure

Every hop, from your log store to a model API to wherever the response lands, is a boundary you now have to account for, and every new integration adds another one. Hosting the model yourself means that infrastructure work lands on your team specifically. SentinelLABS, SentinelOne®’s threat research team, mapped 175,000 exposed Ollama hosts across 130 countries this year, many with tool-calling enabled, meaning code execution reachable from the open internet. That's the scale misconfigured self-hosting already operates at, in the wild, today.

Secrets management, API keys, service credentials, and model weights need their own rotation schedule that nobody enforces but you.

The Maintenance Surface Nobody Sizes Up Front

Every third-party feed, ticketing hook, or notification channel you wire up is an API you now watch indefinitely for breaking changes and schema drift. Permissions, guardrails, and injection defense for your own agent are now your responsibility. The routing logic, prompts, and evaluation harness usually live in one or two people's heads, and when they leave, the ability to safely change any of it leaves with them.

What You Trade Away 

Building your own tooling lets you tune it precisely to your own environment. The argument for a multi-tenant platform is that it improves for every customer by learning from one customer's incident, whereas a system built just for you only ever learns from you. Threat intelligence feeds are the usual attempt to buy back that value. Still, they trail disclosure by days or weeks, built from indicators already known and published, rather than from what another customer's environment just surfaced in real time.
Concretely, this means detection engineering is now a role someone on your team holds, whether or not anyone actually has that skill set. The open-source route is real and free, but it caps your detection quality at whatever the open-source community has already published, well short of what a platform learning from thousands of customers' incidents sees.

Detection engineering is one of several disciplines this plan usually assumes a single person covers. Model selection and evaluation are machine learning problems. Keeping a distributed system online under real alert volume is a production engineering problem. Guardrails, injection defense, and permissioning are their own specialty again. Vendors absorb this work the unglamorous way by staffing each of those functions separately and keeping them staffed. The DIY version usually assumes one very good engineer covers all four, indefinitely, on top of their existing job.

The One That Ends Up in a Postmortem

Availability and uptime, with no SLA to point to but your own. Long-term support is a permanent operating commitment starting day one. Zero Data Retention (ZDR) agreements with every model provider you route through, negotiated and tracked by you. Compliance posture, SOC 2, or whatever regime applies to your industry, becomes an obligation you own instead of something a vendor had largely absorbed for you.

The question that actually surfaces six months from now, usually at the worst possible time: when an incident happens, is your system's own trail forensically sound enough to support the investigation? Will it hold up to a regulator's or a cyber insurer's questions after the fact? Those are answers worth having ready long before an incident forces the question. 

The Slow Leak Underneath Both

Engineering time spent building and maintaining any of this pulls from everything else on the roadmap, and that opportunity cost compounds well past the initial build. The on-call rotation for a system that runs around the clock, the runbook that has to stay current as the system changes underneath it, and the ramp-up time every new engineer needs before they can safely touch routing logic someone else wrote months ago all surface the same way. A slower roadmap, quarter after quarter.

Coverage gaps open up on a similar rhythm, in the moments nobody plans for. Migrating between models usually means running two systems in parallel for as long as the migration takes, with the coordination overhead that entails. Patching requires a maintenance window, and a maintenance window means a gap in monitoring while it's open. A working, maintained system absorbs those gaps as part of the job it's already doing. Building your own means absorbing them yourself, on your own timeline, with your own team, every time.

Why This Argument Refuses to Pick a Model

Notice what's missing from all of that: a claim about which model is best, or what any of this costs. That's deliberate. The frontier will keep moving no matter what gets published this month, so any argument built on "our model beats your model" has a shelf life measured in weeks. The real question a SOC leader has to answer is who owns the complexity of managing that choice, forever, on top of everything else already on their plate.

That's also the honest answer to "but I can do it cheaper with self-hosted models." The price of a token was never the line item anyone forgot. It's usually the only line item DIY math accounts for. Everything in the section above is the remainder of the invoice, and none of it appears on a model provider's pricing page.

It's also why running a single frontier model for every step of an investigation is the wrong shape for the job, regardless of its cost. Most investigative steps don't need frontier-level reasoning, and treating every task as if they do is an architectural problem before it's ever a budget one. Specialized, right-sized models cost less and run faster without losing accuracy on the work that never needed a frontier model in the first place.

The Oversight You Still Have to Build

Even the best frontier models, under ideal conditions, still need a structured governance framework to remain trustworthy over the course of a long-running investigation. SentinelLABS put this to the test with a benchmark that ran leading frontier models through an eight-stage reverse-engineering exercise built from a real malware investigation. Only one model completed every stage, and even that run required a human to define the objective, control what evidence it saw, and hold final authority over the conclusion. The researchers' own term for the strongest outcome was "supervised investigative agency," not autonomous intelligence.

That's oversight of what a model does once it's running. A separate, harder question sits underneath it – whether the weights themselves can be trusted before you ever get to that point. That's its own problem worth its own treatment, and one this series will come back to.

It speaks to what frontier models need to be trustworthy in general, and the implication holds regardless of what you build on. Being impressive in a demo and being safe to run unsupervised for months are different bars. DIY means you're the one building the second bar, alone, with nobody checking your work until something goes wrong.

The Trailhead Sign

Plenty of teams have good reasons to build, and plenty of the considerations above apply, in some form, no matter what you choose. This is the sign at the trailhead: know what's on the trail before you're a mile in.

For most teams, the trail marked "truly DIY" dead-ends into build-with before it reaches full independence. That's still a real choice, worth making deliberately, it's just rarely the clean break from vendors it sounds like.

It's the same blind spot organizations keep hitting with AI more broadly. Mistaking today's friction for tomorrow's safety, when the friction is often the only thing keeping a bigger problem out of sight.

A SOC can't run on vibes, and neither can the decision to build one. Every item above is a real, ongoing commitment with a name attached to it inside your organization, whether you wrote that name down when you started or not. AI changes how much of this work exists and who ends up doing it. It's still human work, whether it happens on your payroll or a vendor's.

Buy, build-with, or truly DIY. Three lanes, and most teams land in one of them by accident rather than by decision. Worth asking: where are you today, and was that a deliberate call or a default?

Answer that honestly now, and the job description doesn't surprise you in six months. Answer it by accident, and it will.

Third Party Disclaimer:

All third-party product names, logos, and brands mentioned in this publication are the property of their respective owners and are for identification purposes only. Use of these names, logos, and brands does not imply affiliation, endorsement, sponsorship, or association with the third-party.

The question has shifted. Security leaders spent several years debating whether AI would reshape security operations. That debate has settled. Now the conversation is about pace. How fast can the foundation be built, and what do organizations that moved early have to show for it?

For the second year, SentinelOne® commissioned 451 Research to survey 611 North American cybersecurity decision-makers and practitioners on the state of security operations strategy. The results confirm what we’ve been building toward, and they surface a finding that should recalibrate how most security leaders sequence their AI investments.

The Returns Didn’t Wait for the Roadmap

Many product roadmaps assume a clear sequence and start with building toward higher maturity first with returns following. The data shows that AI is running ahead of schedule.

Nearly all organizations surveyed (96%) are still operating AI at the earliest maturity levels:

• Level 1: Basic monitoring; triage specialist/alert analyst
• Level 2: More senior triage analyst / basic incident responder and investigator

By most measures, AI adoption in the SOC is still early. And yet, 99% of those same organizations already report improvements in incident response and remediation.

The numbers are consistent. Early-stage AI (chatbots handling initial alert triage, automated tools sorting true positives from noise) is delivering before organizations reach advanced maturity. The gap between where most organizations are and what they are already getting is real and consistent across survey respondents.

Organizations waiting for higher AI maturity before building the supporting infrastructure are running the sequence backward. The returns are available now. The foundation built today determines how far those returns scale.

Platformization Has Reached A Verdict

The organizations accelerating AI adoption are also the ones consolidating onto platforms. A platform-oriented security architecture means moving from siloed, specialized tools to an integrated stack built on a foundation that coordinated AI decision-making can actually run on, and one that lets each new capability compound on the last.

The platformization numbers from this year’s survey are clear. 82% of organizations describe themselves as platform-oriented, a 13-point jump in a single year, and 94% expect to be there within three years.

A common assumption is that platform adoption means replacing specialized tools. The data complicates that picture. The same technologies most frequently deployed as standalone tools (EDR, SIEM, CNAPP) are also the top anchors for integrated platforms. Organizations typically start with one of these and expand outward. What changes is the common data layer that enables coordinated AI decision-making, serving as the connective tissue underneath.

Platformization is not coincidental with AI’s emergence. Agentic AI needs connected, continuously updated data to accurately reason across signals and take autonomous action. Fragmented architectures, where telemetry is siloed and pipelines require manual effort, cannot support AI-driven SOC operations at scale. Platform adoption and AI adoption are converging because AI’s data requirements have made integration a structural necessity.

The survey makes the infrastructure connection an explicit one. The top-cited benefit of investing in a data lake for SecOps is supporting AI-driven SOC workloads and agents. Organizations that built the data foundation early have already cleared the barrier stalling others. Those who haven’t, face a prerequisite gap, and the distance is widening rapidly. Architectural readiness is the variable that determines how far AI investments can scale.

Job Satisfaction Is Rising

Every discussion of AI in the SOC centers on detection and response metrics. This report has those too, but there is a finding that security leaders managing attrition should weigh: analyst burnout is declining.

As AI handles repetitive, high-volume triage work, analysts report rising job satisfaction. The role is shifting away from processing an endless queue and toward investigation, threat hunting, and judgment-intensive work. In a market where SOC analyst turnover remains a persistent operational cost, that shift carries real dollar value.

The analyst role evolves, becoming more strategic and more consequential.

A New Attack Surface

The same AI systems changing how SOCs operate are also creating new targets. Adversaries are already probing AI infrastructure including agents, data pipelines, model endpoints, and the governance gaps that emerge when controls lag behind adoption. The report surfaces this tension clearly: Organizations are deploying AI faster than they are securing it.

An AI agent with misconfigured access or an unmonitored data pipeline is an exposure. Securing the AI infrastructure that powers the SOC is happening alongside deployment, whether organizations have planned for it or not. Those without a clear governance posture are accepting risk that may not be priced into their AI investment case.

The potential of GenAI and agentic AI in the SOC is already being realized. The organizations that capture it fully are those building governance alongside deployment. The platform that runs the Autonomous SOC and the platform that secures it are, increasingly, the same platform.

SentinelOne’s Vision: The Autonomous SOC

Everything the report surfaces, from AI returns arriving before maturity to platform consolidation to the improving analyst experience, points to how these are expressions of the same shift. The foundation that enables early AI returns is the same one that determines how far those returns scale, how capable analysts become, and how well the security of AI itself is governed.

The findings align with how SentinelOne has defined the path to autonomous security operations: A progression from AI-assisted triage at early maturity levels to increasingly autonomous investigation, threat hunting, and response, with humans in strategic and governing roles. The report validates that the market is moving through exactly that sequence. Organizations that understand the architecture behind it (the platform integration, the common data layer, the governance controls) are positioning themselves to capture returns at every stage rather than waiting for the destination.

The full 451 Research report goes further into detail, covering what progression looks like at each maturity level, the specific barriers organizations are encountering, and the data behind each finding in full.

Read the full 451 Research report to learn more about how AI is reshaping cybersecurity.

Third-Party & Intellectual Property Disclaimers

All third-party product names, logos, and brands mentioned in this publication are the property of their respective owners and are for identification purposes only. Use of these names, logos, and brands does not imply affiliation, endorsement, sponsorship, or association with the third party.

This blog may include discussion of unreleased services or features. Any unreleased services or features referenced here are still in development and subject to change. Customers should make their purchase decisions based upon features that are currently available.

AI agents have made their way into virtually every layer of your environment. They run in the apps your employees adopt, on the endpoints where agents execute code, as users with access privileges those agents borrow, and in the cloud workloads that scale them. The platform that you trust to secure your endpoints is already already covering where AI operates today.

Here is the through-line that makes this one problem instead of four. Every AI attack starts as an interaction and ends as an action. A prompt gets manipulated, an agent gets tricked, and the damage lands on a host, reaches into an identity, or moves through the cloud. The tools that treat each surface as a separate product hand you fragments. SentinelOne treats them as one chain.

How SentinelOne Defends the Agentic Stack Today

Employee AI use is where the risk quietly enters. Your people are already using AI tools you never sanctioned, through browser, IDE, and API-connected apps and agentic AI tools. SentinelOne discovers that shadow AI use across browsers, IDEs, and copilots, highlights which tools and models are in play and governs it with policy. It keeps confidential data, PII, and secrets from reaching untrusted models, and it stops prompt injection and jailbreaks aimed at the tools you build. Legacy DLP reads patterns; this reads context, which is the only way to catch an attack aimed at AI systems that behave in a non-deterministic way.

The agent layer is where AI stops advising and starts acting. An employee’s prompt sends text. An agent sends commands, holds credentials, calls APIs, and chain actions without a human approving each step. That makes them non-human identities with standing access. SentinelOne governs that access. It inventories the agents and MCP servers already operating and scores what each one can reach and holds every agent to the privileges its task requires. Then it inspects the tool calls themselves, so an injected instruction gets blocked at the moment it would execute. What gets executed lands in a searchable record, and the same enforcement doubles as a kill switch.

While governance decides what an agent is allowed to do, the endpoint is where you find out what it actually did.

The endpoint is where agents execute. This is the frontier, and where SentinelOne has protected customers for over a decade. Our behavioral engine judges what a process does, not what it claims to be. That is how we caught QUIETVAULT – malware that spins up AI agents in “yolo” mode to exfiltrate secrets to GitHub. It is how we autonomously stopped the LiteLLM supply chain attack, where adversaries weaponized the Claude CLI to install a malicious payload. It is how we surfaced a DLL side-loading attack hidden inside an AI tool installer. Real detections, on the endpoint, today. Agents run on the host, and so do we.

The identity is where a hijacked agent runs next. Picture an employee’s AI coding agent that gets hijacked mid-task. It spawns a shell and reaches for cached credentials and cloud session tokens, trying to stop being a process and start being the user. That pivot to identity is what unlocks lateral movement, and it is where most AI attacks are headed. SentinelOne meets the move. It secures human and non-human identities alike, and seeds the environment with decoy credentials and honeytokens no legitimate user ever touches. The instant the hijacked agent grabs one, the trap trips, and Identity responds by forcing an MFA re-challenge, disabling the account, or isolating the host. Authorization at login is not enough. Access gets validated against behavior and pulled at runtime.

The cloud is where AI workloads scale. Consider an internal AI agent running in a Kubernetes cluster with standing access to a customer database. Security teams keep asking the same question about deployments like this. Where is the model connecting, and who is it talking to? SentinelOne answers with eBPF-native runtime protection that judges how the workload actually behaves, and flags the moment that inference service reaches an endpoint it has never touched before. It covers the control plane the deployment depends on, the secrets it reads, the pipelines it runs through, and the data it can access. Defending the AI you build takes more than watching it, it takes action on the workload in real time.

SentinelOne’s Singularity Platform Advantage

Each of these surfaces matters on its own. What closes the kill chain is following an attack across them without losing it at the handoff. This is where a single platform earns its keep. AI telemetry already streams into the Singularity™ Data Lake, alongside the endpoint data the platform has correlated for years. As identity and cloud signals join that same view, an analyst follows one attack from first prompt to final action, without stitching logs across six tools at two in the morning. A manipulated prompt, the process it spawns, the credential it reaches for, and the cloud resource it targets read as one story rather than six disconnected alerts.

Detection that only watches is observation. Runtime action is protection. When the Singularity Platform acts, autonomous response blocks the execution, rolls back the change, and revokes the access at the point of impact, without a human relaying orders between consoles. This is the difference between whether an attack is stopped or just gets logged.

That is the case for securing AI inside a platform built for autonomous runtime response. We are not adding a console to chase AI, we are extending the one already deployed where your agents run.

Questions to Ask When Assessing Your AI Security Options

When evaluating AI security, ask yourself three things.

• Does the solution protect the endpoint where agents actually execute, or is it a roadmap item?
• When a hijacked agent pivots to credentials and the cloud, does that telemetry land in the same platform, or are you manually connecting dots across three dashboards?
• Can the solution act at the moment of execution, or only tell me what already happened?

SentinelOne protects the surfaces where AI runs today. This includes the apps your employees use, the agents they deploy, the endpoints where agents execute, the identities they borrow, and the cloud where they scale. One platform, built for autonomous response. While AI has changed the attack, it does not have to change your architecture.

See it for yourself. Talk to our team about securing AI across your endpoints, identities, cloud, and the AI apps your employees already use, all from the platform you run today. Contact SentinelOne today.

Third-Party Trademark Disclaimer:

All third-party product names, logos, and brands mentioned in this publication are the property of their respective owners and are for identification purposes only. Use of these names, logos, and brands does not imply affiliation, endorsement, sponsorship, or association with the third-party.