Executive Summary

• Four incidents involving OpenAI, Anthropic, Meta and the UK AI Security Institute (AISI) describe AI agents reaching systems belonging to other organizations without their consent.
• While the causes differ, the consistent factor is the models’ persistence rather than their sophistication, whether as endurance across days of failed attempts or as pivots to entirely new vectors.
• Security teams have traditionally studied the artifacts attackers leave behind, but an agent that simply writes unique, disposable tools makes the model itself the thing worth studying.
• SentinelLABS has been benchmarking frontier models in agent harnesses for months. We observe that the capability that lets GPT-5.6 Sol complete a long-horizon malware investigation is the same one that lets it sustain a two-and-a-half-day intrusion.
• A model may independently determine the methods or targets it uses, but it does not choose its high-level objective or the access it is given to pursue it. We argue that “the AI did it” will not survive contact with the first incident outside a frontier lab.

Four Disclosures, One Pattern

Across four weeks in July and August 2026, OpenAI, Anthropic and Meta have each admitted that their models reached systems belonging to other organizations without consent, and the UK’s AI Security Institute (AISI) published a fourth account describing agents that invented identities and tried to slip a malicious contribution into a live open source project.

The disclosures differ in almost every particular, including whose mistake it was, whether the model defeated a control or simply found one missing, and whether anything was really “escaped” at all. Arguments over those details may run for a while, but the four accounts share something more interesting than their differences, which is that no individual piece of tooling mattered very much.

That observation should sit slightly uncomfortably because most of how intrusions get investigated assumes the opposite.

Early evaluations of LLM cyber capability asked fairly narrow questions. Would a model comply with an obviously malicious request? Did it meaningfully advantage a human attacker? By 2024, the answer was a qualified “yes”. Models could produce serviceable components of an offensive operation, but they could not reliably integrate them into a sustained or adaptive campaign. Beyond short well-defined tasks they became unreliable in ways that made them a liability. The operator was still the operator. The model did scoped work inside a structure someone else maintained.

The disclosures show the boundary between operator and tool is moving faster than the evaluation literature. A capable enough model placed in an agent harness, given tools, memory, permissions and something to achieve, starts absorbing functions that used to be spread across the operator, the toolchain and the payload, which becomes something generated for a single target and then discarded. Kill the process and the agent writes another; block the channel and it improvises around it.

In such incidents, the malicious capability shifts from the code left behind on a machine to the system that produces the next piece of code once the last one fails. In an operational sense, the model is the malware.

Persistence Is the Defining Characteristic

In July, OpenAI agents driven by GPT-5.6 Sol and an unreleased internal research model found a previously unknown vulnerability in a self-hosted Artifactory instance and turned the shared service into a message board allowing agents running different models and evaluations to exchange exploits and coordinate their work. OpenAI disrupted that channel, however the agents recreated it through Artifactory’s remote cache and continued collaborating, eventually breaking out of the evaluation sandbox and onto the public internet. The campaign ultimately compromised Hugging Face’s production infrastructure where it operated for roughly two and a half days.

Hugging Face’s technical reconstruction recovered around 17,600 actions, most of which failed. That is a clear demonstration of the importance of persistence. When confronted with failure, the agent rebuilt its tooling, restored communications when a channel was cut, returned to leads it had abandoned, and kept grinding until a set of otherwise unremarkable weaknesses could be assembled into a workable attack path.

The advantage was not a breakthrough at any one step so much as the relentless effort to explore possibilities a human team would have written off hours earlier, or never reached at all.

While the Hugging Face intrusion demonstrated persistence as endurance—the refusal to stop—other incidents demonstrate persistence as adaptability: the refusal to let a failure in one vector terminate the pursuit of the goal.

Anthropic and Meta have both reported incidents that trace back to the same third-party evaluator, Irregular, whose test environment was left connected to the internet by mistake. When the door was left open, Anthropic’s Opus 4.7, Mythos 5, and an internal research model reached three real organizations, attempted social engineering against real people, and in one case pushed a malicious Python package to PyPI, where it was downloaded and executed on fifteen systems during the hour it stayed up. Meta has also confirmed that one of its models reached and compromised an external firm through the same misconfiguration.

What Anthropic’s three models did once they had some apparent recognition of where they were is instructive. One carried on with the attack, another reasoned it was still in a simulation, and a third stopped. Whatever produced that third outcome, it is not a control anyone should be building a response plan around. A safeguard that behaves three different ways in three attempts, for reasons yet to be explained, is not something an organization can rely on.

A fourth incident reported by the UK AI Security Institute is useful because it takes containment off the table. In this case, internet access had been granted deliberately, in line with the institute’s standard testing procedure, so there was no boundary to defeat and no misconfiguration to blame afterwards.

The agents, built on Mythos 5 and GPT-5.6 Sol, were being assessed on cyber capability and set about it in ways their operators had not anticipated. They selected a real open source project as a target, researched the people who maintain it, and built fabricated identities to deceive them.

When the direct technical route to the objective proved unpromising, the work shifted onto the humans around the codebase, a fairly ordinary supply chain playbook, albeit one that was executed without any explicit instruction to do so.

Attempts were also made to influence other AI coding systems in the project’s orbit, on the assumption that a contribution reviewed by a model faces a different kind of scrutiny than one reviewed by a person. Human review caught the most serious activity, but those wins are increasingly tenuous in a world where developers are growing comfortable with trusting LLM generated code by default.

One incident crossed a real boundary, two wandered through a misconfigured one, and the last had no boundary to cross. Different stories, but they lead to the implication that failure has stopped functioning as a constraint, and “raising the cost” on an attacker is no longer a viable security posture.

The Same Capability, Pointed the Other Way

We have spent much of 2026 running the same models in agent harnesses against real work. Our most recent benchmark rebuilt our investigation of fast16, a 2005 sabotage implant, into an eight-stage autonomous reverse-engineering task, run in our own environment against a benign objective with observation throughout. GPT-5.6 Sol was the only publicly available model to finish it, a result worth pairing with the fact that GPT-5.6 Sol was one of the models that compromised Hugging Face.

Every cohort we ran produced sound technical insight, so insight was never what separated the runs that finished from the runs that stalled. The difference showed up in what we called project-scale recovery, meaning the ability to withdraw a claim once new evidence contradicted it, work out which conclusions and artifacts depended on the discarded result, carry the correction into the affected files, and then reopen the whole thing and run a check capable of disproving the corrected version.

That description doubles as a summary of the Hugging Face timeline. An agent able to abandon a failed approach, establish what else it invalidates, rebuild the tooling that depended on it and carry on without losing the thread is doing in somebody else’s Kubernetes cluster what ours were doing in an IDA database. When our team first saw this incident we did not assume the models had “gone rogue”; the behavior looked similar to other problem-solving approaches we had seen in our own testing.

An Object Becomes a Behavior

None of this should feel entirely unfamiliar to defenders. Two earlier shifts in adversary behavior, initial-access brokerage and Living off the Land, had already pushed security away from an artifact-centric view of malware and toward a behavioral understanding of adversary operations. To understand the emerging threat of agentic systems we should examine the successes and challenges with these well-known abuse paths.

Initial access brokers split compromise from exploitation. One adversary would steal credentials or breach an edge device, and then sell that foothold to whoever had the skills and the motive to use it. Europol wrote about the resulting crime-as-a-service economy over a decade ago, and its 2025 assessment shows the same trade still going strong.

The defensive response was to stop treating access as an event and start treating it as a lifecycle. This shift in philosophy is what accounts for many of the exhausting reauthentication flows now built into enterprise environments. That control raised costs for users and attackers alike and it diminished the value of a stolen credential in the marketplace. Agentic systems remove those costs for attackers as discovery, exploitation, lateral movement and whatever the attacker actually came for can happen in the same loop as the credential theft.

This leads us to our second challenge, the rise of Living off the Land techniques where attackers traded their own malware for administrative tooling already installed on the machine. Here attackers traded capability for cover, since every tool an attacker brings with them is another chance for the defense to spot the intrusion or tie it to a previous attack.

Agents take that logic off the host entirely, Living off the Land, the cloud and the open internet at once, and writing whatever they need from scratch when the tools they need do not already exist. Command and control for the Hugging Face intrusion ran over pastebins, request-capture services, and file-drop sites. None of the infrastructure used in the compromise belonged to anyone under attack.

Both of these shifts moved defense towards behavior and away from objects. What remains untested is whether the controls we built for adversary behavior ten years ago still hold up when the behavior arrives as thousands of individually boring actions, sequenced differently in every attack and at a tempo no human operator can sustain.

An agent’s ability to persist in a relentless attack revolves around identity and authority. The questions worth asking are about sequence rather than artifact: what chain of actions is running, which identity and authority connect them, at what point did behavior exceed the role it was granted, and how quickly can that authority be pulled? We are going to need a lot of testing to ensure that the current gaps in our infrastructure don’t become chasms.

The Debt Was Always Going to Be Called In

Speaking of chasms, much of the software organizations depend on will be expensive to change. It may have accumulated over decades, picked up dependencies, or can only be touched during narrow maintenance windows, if at all. That debt has largely been survivable because vulnerability discovery was scarce. Finding a flaw took expertise and time, building a reliable exploit took more, and organizations could carry vulnerable software for years knowing only a fraction of its weaknesses would ever receive sustained attention from anyone.

Sustained attention is the one thing an agent has in abundance, and surfaces that were quietly secured by obscurity are now low-hanging fruit for agentic models. The same capability cuts the other way, which is why the NCSC has been warning about a “vulnerability patch wave” as AI starts surfacing debt across the stack. However, ownership, deployment and verification remain human, and costly, work. Maintainers cannot review unlimited contributions, enterprises cannot manufacture maintenance windows, and OT cannot go offline every time a model finds a vulnerability that threatens the water in our pipes or the electricity in our lines.

Worse still, there is nothing orderly about the way technical debt comes due. It gets settled during an actual intrusion, at the point where the rate of exploitation outruns the rate that the system’s defense can respond. Whether agentic attackers have already crossed that line is a fair question. The four disclosed incidents from July and August 2026 are a small and biased sample: All involved organizations that log heavily and had every reason to scrutinize model behavior. The most troubling incidents will likely occur in organizations that lack the capability to do either.

What, then, can organizations do? The usual advice still applies. Work out which debt can turn into an incident, pay down the expensive parts first, and wall off what cannot be fixed yet. However, the most important change that an organization can make is the ability to absorb change, which means automated testing, hot patching, and an engineering culture where making changes to systems is routine rather than an event.

AI will help with porting old code and proposing fixes, and it will also grow codebases well past the point where anyone can keep track of them. Writing code faster than attackers or relying on larger token budgets cannot be the answer. The imperative has to be reducing the amount of critical software that nobody feels comfortable touching.

“The AI Did It” Is Not an Accountability Model

A version of this story in which the agent is the protagonist is already circulating, and it is worth resisting for reasons that follow directly from the argument above. Naming the model as the malware is meant to deny it a motive, not hand it one, since malware is something defenders study and contain while accountability stays with whoever deployed it. We argue that “the AI did it” will not survive contact with the first incident outside a frontier lab. While a model may independently determine the methods or targets it uses, it does not choose its high-level objective or the access it is granted to pursue it.

Hugging Face reconstructed 17,600 actions after the fact. Anthropic has logs that reveal which models kept going and which one stopped. OpenAI has the agent traces that describe how the model reasoned its way into conducting the attack. Very few of the organizations now putting agents into production could produce such an account of their own systems, and in practice that gap is the accountability argument. Our own benchmark runs generated more than 23 billion tokens of logged activity, which is a fair indication of what it costs simply to determine after the fact what an agent did.

Anyone deploying an agent should be able to answer three questions about it before an incident rather than during one: what sequence of actions it took, whose identity and authority it used to take them, and how quickly that authority can be withdrawn.

Those questions were answerable at the frontier labs because observation was the point of the exercise. Everywhere else they are a deliberate investment, and one that has to be made while the agent is still useful rather than after an incident makes it necessary.

Executive Summary

• SentinelLABS developed a multi-stage reverse-engineering benchmark for the latest generation of frontier models by recreating our recent investigation of fast16, a unique 2005 sabotage implant.
• Most AI benchmarks test bounded tasks. This benchmark tests whether a model can keep a malware investigation trustworthy as new evidence repeatedly invalidates its earlier conclusions.
• OpenAI’s GPT-5.6 Sol was the only publicly available model to complete the full eight-stage investigation, giving concrete shape to what ‘Frontier-class’ capabilities offer analysts. GPT-5.5, GLM-5.2, and the Opus 4.x family produced capable local analysis but could not carry it through the gradient.
• What distinguished the completed runs was project-scale recovery: withdrawing contradicted conclusions, repairing technical artifacts, and updating dependent reporting without losing the investigation.
• Senior reverse engineers remain essential. Even the strongest runs made semantic errors, accepted weak quality controls, and claimed readiness prematurely. We assess the best current use as supervised investigative agency, with human analysts defining objectives, exposing blind spots, and retaining final publication authority.

Beyond Vulnerability Discovery

Since ChatGPT arrived in late 2022, we have been bullish on what large language models could do for reverse engineering and malware analysis. The early models were useful for teaching but too rudimentary for production work; that changed with the advent of reasoning models. OpenAI’s o1-preview, in September 2024, was the first to show the kind of sustained problem-solving the work demands, and within months Sean Heelan had used o3 to find a net-new vulnerability in the Linux kernel. In cybersecurity, though, our understanding of what these models can do remains stovepiped to vulnerability discovery.

The frontier labs took on vulnerability discovery deliberately, because that competency keeps agentic code generation from quietly shipping vulnerable code at scale. OpenAI built Aardvark, since folded into Codex; Google DeepMind announced Big Sleep, available internally to its Project Zero researchers; and Anthropic followed with selective access to Mythos Preview. Concerns that these capabilities could be misused have led the labs to stricter guardrails and ‘know your customer’ style controls that limit access to specific capabilities, or to entire model variants.

OpenAI’s Daybreak initiative and its Trusted Access Program opened a dedicated variant, GPT-5.*-cyber-preview, with guardrails relaxed for cybersecurity use cases, while Anthropic’s Glasswing initiative and its Cyber Verification Program provided early access to Mythos Preview and the promise of lesser guardrails respectively. For a short period in mid-June 2026 access to the highest-end flagship models from both providers required some form of U.S. government clearance. At the time of writing, GPT 5.6 Sol is widely available, while Mythos 5 still requires clearance and access as a Glasswing partner.

The existence of this new class of models left us with an unusual task: benchmarking what these models can actually do on the work defenders care about, and assessing whether they live up to the surrounding hype. If they do, we have to reckon with what that means for malware analysis and reverse engineering, disciplines that until now have been limited mostly by how little expertise exists relative to the collective need.

A Benchmark Built From a Real Investigation

We recently published our research on fast16, a 2005 Windows toolkit built to sabotage high-precision solvers used to model nuclear-weapons behavior. The sample provided an ideal test case because its layered design punishes shallow analysis.

On the surface, svcmgmt.exe appears to be a Windows service implant with two embedded components. Their combined logic becomes visible only through a higher layer of abstraction, an encrypted Lua-driven operations framework. A kernel driver easily mistaken for a filesystem rootkit actually contains a patching engine with 101 rules, and those rules must be repurposed to identify candidate target software before an analyst can extrapolate the approximate effects of the sabotage operation. This investigation is a daunting task for human experts, and reproducing it autonomously should be considered a significant milestone for artificial intelligence.

When Vitaly Kamluk shared our discovery of fast16 at Black Hat Asia, he pointed to the failed attempts of Opus 4.6, GPT-5.4, Grok 4.2 Reasoning, Gemini 3.1 Pro, and DeepSeek Reasoner to triage the sample autonomously. At best, the models concluded fast16 was a rootkit and could not move past that assumption, converging on the same plausible but inaccurate conclusion most human reverse engineers reached on first contact.

Used instead as assistants, the models earned their keep. Opus 4.6 and GPT-5.4 compressed what was originally a three-week dedicated expert effort by Kamluk into an 8-10 hour task manageable by a less experienced reverse engineer. Autonomous investigation was out of reach when we began this work; co-intelligence was the ceiling.

Standardizing the Benchmark

We codified our reverse-engineering methodology, inspired by Rolf Rolles’ approach to statically analyzing binaries, into a skill the harness loads on /reverse, which removes prompting variability across models. Each run gives the model a sample, IDA Pro 9.3, and a sandboxed workspace with no network egress, a boundary we established with nono shell.

The model must produce fully annotated IDA databases by renaming variables, recasting data types, categorizing functions, recovering hidden components, and folding those findings back into the host-sample database.

Only after producing a gold-master IDB for approval does a run advance to walking the execution flow and detailing findings, with every claim anchored to specific addresses and artifacts so the work can be independently verified. Judging the model on a refined IDB rather than a prose report closes off the deductive shortcuts and approximations typical of LLM malware triage.

We took an ecological approach, a bounded case study that benchmarks a capability profile in a complex real-world scenario rather than feeding another general leaderboard or isolating a single narrow competency. We also deliberately tested the combination of model and harness, since agentic workflows are how practitioners actually use these systems, and most benchmarks still test models in abstraction.

Runs level up through eight stages, each introducing new artifacts and minimal prompting for the new task:

Measuring ‘Intelligence’

When Nick Fountain interviewed us for NPR’s Planet Money, he observed that fast16’s brand of sabotage amounts to epistemic warfare, an attack on the target’s confidence in its own ability to know something to be true. There is a fitting symmetry in using the same sample to examine what the frontier labs sell as intelligence, because the difficulty there is also epistemic. ‘Intelligence’ as a concept bundles many interwoven capacities, and model tiers simplistically priced as if intelligence gets cranked up with a simple knob tells us nothing about what any one offering is worth on real work.

Many of those capacities already have familiar names, and the names do finer work than a score. Take thoroughness, the completeness of a particular piece of work. An analyst who consistently applies that effort where it matters is diligent. One thorough artifact does not make a diligent analyst; diligence is the longitudinal quality, and thoroughness one of its observable expressions.

Exhaustive coverage is not diligence either. An analyst who inspects everything indiscriminately while neglecting the consequential work is being pedantic, and their effort is wasted on the broader investigation. Diligence, in short, is sustained, priority-sensitive thoroughness.

Many similar qualities sit inside what we call intelligence, and a meaningful benchmark must account for them as we would with a human analyst, by characterizing behavior across an entire investigation rather than compressing it into a rank.

We scored two different things. Local analytical quality asks whether the model made technically sensible decisions with the evidence available. Investigative horizon asks how long it preserved that quality as the scope, evidence, and reporting obligations expanded. The two are related but not the same, and the gap between them is where most models failed.

What the Runs Showed

GPT-5.5 produced long autonomous runs, sometimes grinding uninterrupted for a day or two, with results relatively consistent across reasoning-effort settings. It did meaningful, semantically conservative work refining IDBs and repeatedly rejected edits that improved a count while making the decompilation less truthful. Its weakness was aggregation. Residual cast debt, function-tree defects, or incomplete cross-component verification kept it from clearing broad evaluation gates, and it never converged its extensive work into a complete package for review. Notably, nominal effort settings and multi-agent use showed no monotonic relationship to clearance. GPT-5.5 never escaped the grind phase of level 1.

We ran a single GLM-5.2 run through a serverless deployment on Fireworks AI with a $100 budget. The run showed measured local experimentation and capable backup-based recovery, which it needed after the most severe promotion failure we observed, losing 113 embedded functions and having to restore them before working again. One run is not a characterization of the model, but it is enough to temper claims of GLM-5.2 performing at the level of the frontier completers.

Opus 4.7 and 4.8 advanced further, showing capable technical correction and evidence-responsive revision, including a strong example of converting a speculative relationship into a structural test and rejecting it. Their recurring limitation was milestone overpromotion, a familiar eagerness to convert favorable local results into readiness claims before visible defects were resolved. They required operator challenge at important quality boundaries and did not sustain closure through the full benchmark. We are not altogether certain these newer iterations are in fact performing better than the earlier Opus 4.6 for our use cases.

The common pattern across this tranche was local competence without dependable project-wide closure. These models performed many of the individual behaviors seen in the strongest runs. What they did not demonstrate was the same behavior recurring through the full eight-stage investigation.

GPT-5.6 Sol | A New Class of Contender

When GPT-5.6 Sol was introduced, we seized the opportunity to evaluate it against our real-world benchmark. As the name implies, we expected incremental improvement from GPT-5.5 but instead found the model operating in a league of its own, with all three GPT-5.6 Sol runs completing the benchmark at different effort levels.

Sol was far from infallible. Its runs declared checkpoints clean or complete before later review found meaningful problems. The distinctive pattern was what happened next. Sol repeatedly responded to a local failure with a structural fix, reassessing the quality-control process that had let the mistake through and building scaffolding, such as codifying file hashes for even routine operations, to keep entire classes of failure from recurring.

The xhigh run provides the clearest example because its error began with a valid observation. fast16 contains a program of 101 byte-pattern rules that recognize locations inside an executable, capture addresses, replace code, and repair the executable metadata those changes require. Replaying that rule program across a larger software collection, Sol found that some rules fired not only in LS-DYNA, the engineering package fast16 targets, but also in MOHID and PKPM, two applications built from a similar Fortran lineage. It proposed that fast16 might be a general compatibility component rather than a tool engineered for sabotage.

The collisions were real, and we made similar observations early in our own investigation, but the explanation gave too much weight to the mere presence of matches. The model corrected course by demanding greater rigor of itself, asking whether the matches produced a meaningful modification sequence. In LS-DYNA they did. In MOHID and PKPM they produced only haphazard fragments, often triggered by generic compiler-generated code.

What first looked like broad compatibility was better explained as collateral risk from loose rules. Sol not only self-corrected but found better footing for that correction than we did. It then withdrew the compatibility hypothesis, preserved the valid measurements, restored the LS-DYNA modification mechanism as the authoritative account, and marked the rejected branch as historical provenance so later analysis would not inherit it.

5.6 Sol’s performance has pushed the boundary of the possible for automation in cyber threat investigations. Given its ready availability, we are interested in seeing how these capabilities enable the practitioner ecosystem to advance the state of the art in the same manner as alternate research disciplines like math and physics.

Project-Scale Recovery

What separated the completed runs from the rest was not technical insight, error detection, or willingness to revise. Every cohort supplied examples of each. The recurring difference was the scale at which recovery occurred. Across a long-horizon investigation, the completed runs did more than correct the sentence or script in front of them. They rebuilt the part of the project that later work would trust, in a repeatable pattern:

1. Withdraw the contradicted claim rather than softening its wording while continuing to rely on it.
2. Map the blast radius, identifying every conclusion, artifact, and test that depends on the disputed result.
3. Repair the cause, correcting the analysis or the quality-control gap that let the error through.
4. Carry the correction through the files another analyst or later stage will actually use.
5. Reopen the artifacts and run a check capable of disproving the corrected claim.
6. Separate work that blocks the current conclusion from uncertainty that can be disclosed and deferred.

That last step is easy to underestimate. An investigation can fail by stopping too early, and it can also fail by expanding forever. Deciding which unresolved questions block a conclusion is what convergence means in this benchmark.

This is not a single new reasoning faculty. It is the coordination of familiar capabilities over time, spanning correction, scope control, verification, state management, synthesis, and stopping judgment. The evidence supports project-scale recovery as a common feature of the completed runs. It does not prove a permanent intelligence threshold, and looking only at finishers cannot explain why every other trajectory fell short. The number of runs per family is small, and operator involvement was consequential throughout. What survives those caveats is the modest formulation that the completed runs repeatedly re-established a trustworthy investigative state after later evidence destabilized it.

Notes on Cost

Across all experiments, our runs consumed more than 23 billion tokens across model families and effort levels. The aggregate cache read rate was 93.58%, translating to different discounting rates across providers. Our self-hosted GLM-5.2 run offers a floor for comparison, producing 267.41M tokens on a $100 budget and a decent level-1 attempt charitably comparable to some GPT-5.5 and Opus 4.7 performance.

What These Results Mean for Analysts

The practical lesson stops well short of replacement for senior reverse engineers. The strongest runs still made serious semantic errors, accepted weak quality controls, promoted work too early, and depended on operator review at consequential moments. The new capability is that some runs remained useful after those failures. They owned substantial investigative workstreams, recovered from invalidated theories, preserved the valid parts of earlier work, and continued into publication-oriented analysis without starting over.

That changes the role these systems can plausibly play. A weaker model may be valuable as a local copilot, explaining a function, testing a hypothesis, or drafting a section. The strongest observed runs could act more like supervised investigative agents, maintaining and repairing a larger body of work across days, artifacts, and changing evidence. The supervision qualifier remains essential. Humans defined the investigative objective, controlled evidence release, exposed important blind spots, decided what quality meant, and retained final publication authority.

The scale of the shift deserves emphasis. Reverse engineering and malware analysis have been kept from scaling for three decades by the scarcity of expert practitioners. That constraint does not disappear, since expert guidance still determines whether the work can be trusted, but a single expert supervising these systems can now expand their output by any metric we track.

Conclusion

The line the benchmark exposed ran between local competence and sustained ownership of an investigation, rather than between smarter and lesser models. GPT-5.5, GLM-5.2, and the Opus 4.x family frequently produced valuable analysis. GPT-5.6 Sol also made meaningful errors. What distinguished the completed runs was their repeated ability to recover and diffuse that correction throughout the whole project, rather than just confronting their immediate context.

Having completed this stage of our evaluations, we took the best candidate produced by 5.6 Sol High and put it through an extensive adversarial refinement process using an ensemble of models to produce the highest automatable quality of IDBs and investigation artifacts. We are sharing these so other researchers can verify the work output against the sample and build on them rather than start from the raw binary. We make no claim that it is perfect, only that it is the greatest level of automated refinement the most capable models achieved through proper methodology, repeated standard enforcement, and a lot of tokens burned.

That is a less dramatic claim than a new form of autonomous intelligence. It is also more useful. For defenders, the question that matters most is whether the models can reliably preserve and advance a trustworthy investigation. Once we can count on that, it’s up to us to redefine a greater scope of impact for a new frontier of cyber threat investigations.

Resources

fast16-idbs – Gold-Master IDA Databases

In April, SentinelLABS’ Tom Hegel published an initial assessment of the first five weeks of the conflict. Three months later, the evidence supports refinement.

Executive Summary

• The cyber risk remains quieter than the public narrative. It rests on persistent access, trusted administration, service-provider pathways, selective disruption, and personas that magnify technical effects.
• Iran-linked activity is not a single threat set. MOIS, the IRGC Intelligence Organization, the IRGC Cyber-Electronic Command, personas, surveillance operators, and opportunists pursue distinct missions.
• The principal strategic risk is access optionality. The same compromised account, service provider, or remote-management foothold can support intelligence collection, downstream targeting, or selective disruption as tasking changes.
• MOIS-linked personas such as Handala, Homeland Justice, and Karma combine intrusion, destruction, disclosure, and coercion. Their impact claims frequently outpace independently verified evidence.
• OT risk remains exposure-driven. Internet-facing PLCs, weak credentials, and poor remote-access governance have enabled real disruption, but interface access alone does not demonstrate process manipulation or physical effect.
• Inside Iran, shared-service concentration, connectivity controls, and limited disclosure obscure the incident picture. External operations, domestic control, and resilience failures intersect there.

A Working Taxonomy

“Iran-linked” is a broad analytic descriptor, not a single actor or command structure. Iran’s cyber ecosystem spans operators tied to the Ministry of Intelligence and Security (MOIS), the IRGC Intelligence Organization, and the IRGC Cyber-Electronic Command, plus state-aligned collectives, domestic-surveillance clusters, and opportunists. These entities differ in command relationships, missions, targeting, tradecraft, and risk tolerance.

Public naming adds complexity. Vendors assign different labels to overlapping activity sets, and some names refer to actor clusters while others describe campaigns, personas, malware families, or infrastructure.

We offer the following as a working crosswalk, rather than a claim that every label is a one-to-one alias, that every organizational relationship is proven, or that the picture is static.

We base our distinction on mission rather than branding. In practice, we can identify several recurring mission sets:

• persistent espionage and access enablement;
• destructive, coercive, and influence operations through public personas;
• high-trust social engineering and cloud compromise;
• surveillance of dissidents and civil society; and
• opportunistic targeting of operational technology.

These categories overlap, but they offer a more reliable basis than actor names for assessing intent and prioritizing defenses.

1. The Durable Threat Is Access Optionality

Three months of headlines have tracked leaks, defacements, and outages. However, there is a quieter accumulation of access that matters more, where a foothold gained for collection today can be converted to disruption tomorrow or whenever tasking changes.

For example, activity attributed to the MOIS-linked Seedworm/MuddyWater cluster began in early February, before the opening strikes. Affected environments included a U.S. bank, a U.S. airport, nonprofits, and the Israeli operation of a U.S. software supplier serving defense and aerospace customers. Researchers identified multiple backdoors and an attempted transfer of data to commercial cloud storage.

Because that activity preceded the kinetic campaign, it is tempting to call every intrusion wartime pre-positioning. In practice, the evidence supports a narrower reading. We might know access was gained before the strategic picture changed, but that doesn’t tell us what the compromise was originally for. A compromised supplier may expose customer identities and trusted administrative paths, but it’s only a software supply-chain compromise if there is evidence that downstream customers, builds, updates, or distribution were affected. What matters is that the operator had options once the conflict escalated.

A similar timing pattern appears in Screening Serpens. Unit 42 identified six new RAT variants deployed between February and April against apparent targets in the United States, Israel, the UAE, and the wider Middle East. The campaigns continued the actor’s tailored recruitment lures while adding AppDomainManager hijacking, and the conflict appears to have increased tempo without creating the espionage mission.

This is also why high-trust people are part of the enterprise perimeter. Long-running APT42 operations have targeted journalists, researchers, NGOs, academics, activists, and government-linked individuals. One compromised cloud account holds organizational context, relationships, and internal deliberations, and can yield collection, impersonation, lateral targeting, and entry into the wider organization.

Third party reporting extends that logic to service providers. Check Point’s July reporting on Cavern Manticore described intrusions in which existing RMM access and compromised IT-provider environments opened paths into targets. Check Point assesses an MOIS relationship based on technical and operational overlap with MuddyWater and Lyceum; public corroboration remains limited, so the mapping should remain moderate-confidence.

SysAid itself was not compromised, and no SysAid vulnerability was involved. The actor already had access and abused a legitimate deployment feature, and in many intrusions the weakness is the authority already granted to an administrator, service account, RMM agent, identity provider, or support organization rather than the product itself.

The war has also not displaced Iran’s standing regional requirements. Previously documented APT34 activity against Iraqi government infrastructure shows a durable mission of collecting political, diplomatic, and telecommunications intelligence from neighboring states. Iraq and the Gulf are not peripheral theaters simply because the visible strikes occur elsewhere.

Our assessment is that Iran’s most valuable cyber asset is optionality. Persistent access can support immediate collection, future tasking, transfer to another operational element, or disruption when political value exceeds operational cost.

At least one recent Handala wiping script was assessed as likely AI-assisted, and Iranian operators are likely using generative systems to accelerate coding, translation, lure development, and impersonation. The evidence supports an efficiency multiplier rather than a separate mission set.

2. Persona Operations Are Operational Infrastructure

Iran-linked intrusions are often claimed in public by personas with hacktivist-style brand names like Handala Hack Team or Homeland Justice. We observe that these personas provide threat actors with more than just post-intrusion propaganda. They serve as reusable operational infrastructure, supporting attribution masking, coercion, disclosure, intimidation, and amplifying claims.

A meaningful portion of higher-impact public activity resolves to personas of the MOIS-linked Void Manticore apparatus. In March, the U.S. Department of Justice seized four domains associated with Handala, Homeland Justice, KarmaBelow80, and a related “Red Wanted” operation, identifying shared leak infrastructure, Iranian IP ranges, and a common playbook combining destructive intrusions with data publication, doxxing, and threats.

That evidence supports treating Handala Hack Team, Homeland Justice, and Karma as related fronts within a common MOIS-linked system, and Check Point has also documented collaboration involving Scarred Manticore. Treating the personas as interchangeable aliases in a strict CTI sense goes further than the evidence allows. Personas can serve different geographic, linguistic, or operational purposes, and the same apparatus may retire, merge, or repurpose brands as requirements change.

It is worth noting that following the March 2026 infrastructure seizure(s), Both Handala and Homeland Justice personas have continued to establish new infrastructure, and communications channels for influence and narrative control purposes. Handala, in particular, has been consistent with regards to communication over the observed timeline.

A more defensible model has three layers:

• State-managed persona systems combine intrusion, destructive effects, data theft, leaks, threats, and messaging under deniable brands.
• Ideologically aligned networks may coordinate targets, amplify claims, or share stolen material without evidence of direct tasking.
• Opportunists and service sellers enter for attention, access sales, or revenue, relying on DDoS-for-hire, recycled data, and low-impact defacement.

These layers interact. An opportunist may sell access to a state operator, and a persona may amplify a third party’s claim, none of which proves shared organization. Treating direction, alignment, collaboration, brokerage, and amplification as synonyms inflates actor counts and obscures the operators with real access.

The March attack on Stryker is the clearest public case for this layer. Stryker confirmed global disruption to its Microsoft environment affecting ordering, manufacturing, and shipping. Early statements said no malware had been detected; a later update clarified that the actor used a malicious file to execute commands and conceal activity, though the file could not spread inside or outside Stryker’s environment.

Handala claimed responsibility, and the Justice Department later stated an MOIS-controlled Handala domain was used to claim the March 11 destructive attack. Stryker did not attribute the incident to Handala or validate the actor’s quantitative claims of device destruction and data loss. Those remain actor assertions rather than confirmed findings.

The public claim is part of the operation. Victim lists, leak samples, countdowns, doxxing, and unverifiable impact figures impose reputational and psychological costs before technical scoping is complete. These are all fundamental steps in the personas’ playbook to help craft and control the narrative and potentially sway the sentiment of the public.  In a persona-led incident, publication and personal targeting may begin while responders are still establishing blast radius.

3. OT Risk Is Real, and Evidence Quality Matters

The strongest public evidence of wartime Iran-linked activity against U.S. operational technology is the April 7 joint advisory on internet-facing Rockwell Automation and Allen-Bradley PLCs, documenting activity against government facilities, water and wastewater organizations, and energy environments, in several cases with operational disruption and financial loss.

Two U.S. cases show why cyber-physical reporting needs exact boundaries. On April 7, CISA, the FBI, NSA, EPA, Energy Department, and U.S. Cyber Command warned that Iran-affiliated actors were exploiting internet-facing Rockwell Automation/Allen-Bradley programmable logic controllers. The agencies reported manipulation of project files and HMI/SCADA displays, operational disruption, and financial loss across multiple critical-infrastructure sectors. That is a confirmed cyber-physical campaign described by multiple government agencies.

The June California Water Service case is narrower. Handala claimed that it had hacked the utility and could have interrupted water service. Analysis of the actor’s published material supported access to customer billing data and an internal RTKBase/NTRIP GPS-correction environment used by field crews. Cal Water said it was investigating and that preliminary findings showed no known operational disruption to water, wastewater, or billing. The supported finding is potentially significant IT and field-support-system access—not demonstrated control of treatment, chemical dosing, or water distribution.

We should be cautious about attribution. The advisory described Iranian-affiliated cyber actors and referenced earlier IRGC Cyber-Electronic Command activity associated with CyberAv3ngers. Separately, Unit 42 maps the cluster it tracks as CL-STA-1128 to CyberAv3ngers and Microsoft’s Storm-0784 label. Those assessments are compatible, but they do not prove that every exposed PLC or claimed compromise involved the same operator.

Pro-Iran channels keep publishing images and videos appearing to show access to HMIs, SCADA interfaces, and industrial engineering software. Some may be genuine unauthorized access, but others may not. Analysts should apply an evidence ladder, in which each rung requires evidence beyond the one before it.

1. Provenance and target validation: Can the organization, system, location, and timeframe be independently established?
2. Interface visibility: Can the actor display a login page, dashboard, or management interface?
3. Authenticated interaction: Can the actor navigate the live interface or query current values?
4. Write or control capability: Can the actor change a setting, logic file, operating mode, or command value?
5. Process effect: Did the change alter an operational process outside the interface?
6. Physical or safety consequence: Did the process change produce a verified real-world outcome?

Stronger claims need logs, process data, engineering review, operator testimony, timestamps, configuration evidence, or independent confirmation of the outcome. These personas will falsify or embellish claims at times for various reasons.  It is crucial that we scrutinize claims for validity, accuracy and technical viability.

Regardless of which actor is behind any given incident, the attack surface is well-documented: Internet-facing PLCs, unmanaged HMIs, remote engineering services, default credentials, permissive vendor access, and poorly segmented management networks. Any of these can create opportunities for meaningful disruption. In some environments, what separates a low-skill intrusion from a serious incident is the authority exposed to the internet rather than attacker capability.

Evidence quality should determine how an incident is described, not whether an exposed control path is closed.

4. Inside Iran | Concentration Risk, Underreporting, and Surveillance

Iran is also on the receiving end of this conflict, and the picture inside the country is shaped by more than just foreign intrusions. Shared-service concentration, connectivity controls, domestic surveillance, and weak disclosure increase both operational risk and analytic uncertainty.

Banking: The Dependency Is The Strategic Finding

In this context, Iran has experienced at least two officially reported banking disruptions this past June, but public record does not establish these as a one continuous intrusion.

On June 14, Iranian authorities reported that an attack on shared communications infrastructure disrupted services at four banks: Bank Melli, Bank Tejarat, Bank Saderat, and the Export Development Bank of Iran, with no unauthorized access to or deletion of customer data identified. A second reported attack on June 23 disrupted card services at three of the same banks, affecting ATMs, point-of-sale terminals, and mobile applications.  These overlaps are notable, but timing and claims alone are not enough to prove a coordinated, multi-stage campaign by any one specific actor.

The incidents remain publicly unattributed. Predatory Sparrow is an obvious comparison given prior high-impact operations against Iranian financial targets, but on the available evidence it is not an attribution. The June incidents lacked the public claim, evidence package, and established destructive signature of the group’s better-known operations.

The strongest conclusion is architectural rather than actor-centric. Failure of a shared communications provider, card platform, identity service, or recovery environment can propagate across institutions and become a national public-confidence event, with outage scale reflecting dependency concentration as much as attacker sophistication.

Fallback systems belong to the same risk model, and the lesson is not confined to Iran. A recovery platform is more than an emergency copy of production. Isolation, capacity, data currency, and administrative security determine whether it operates safely under pressure, and a fallback sharing credentials, management tooling, or upstream providers with production may reproduce the failure when needed most.

Surveillance Infrastructure Can Become Targeting Infrastructure

Iran’s surveillance and communications systems are not separate from the conflict. Associated Press reporting, based on intelligence and operational sources, described Israeli access to Iranian surveillance-camera networks supporting the tracking of senior leadership; absent disclosed access paths or technical artifacts, those details should remain described as reported rather than confirmed.

The structural risk outlasts any single account. Camera networks, subscriber records, location histories, and identity databases concentrate information about people and movement, and infrastructure built to monitor a population can become targeting infrastructure for a foreign service.

The relationship also runs in reverse. Disrupting the same communications layer degrades public warning, incident response, and independent reporting, making it a surveillance asset, an intelligence target, a resilience dependency, and a domestic-control mechanism at once.

Public Reporting Is A Floor, Not A Denominator

Banks are the best-documented internal target set in the public record reviewed here, and not necessarily the only one. Incidents affecting telecommunications, energy, or military-adjacent environments may be underrepresented, a collection hypothesis rather than a confirmed count. Iranian reporting is constrained and fragmented, and incidents may be kept private or described too vaguely to distinguish attack from technical failure. The public incident set is a floor, evidence that at least those events occurred rather than a denominator for total activity.

The same logic cuts the other way, though. Knowing that incidents go unreported is not a license to fill the record with rumor, and however many compromises are plausibly hidden inside Iran, they cannot be used to attribute a specific outage, validate an actor’s claim, or turn an unexplained failure into evidence of cyberattack. When collection is thin, the honest answer is explicit uncertainty rather than false precision.

The Blackout Is Both A Collection Gap And An Attack Surface

Iran began restoring international connectivity after an 88-day shutdown, but the return was partial and uneven. A 2026 technical paper found that forwarding-plane null-routing could leave BGP announcements apparently stable, causing route-based monitors to understate the scale of disconnection.

For CTI, telemetry loss is not uniform. External visibility falls while selected domestic services stay reachable, so apparent recovery in one dataset may reflect exemptions or measurement artifacts. The blackout is also not an all-purpose causal explanation. Public evidence does not establish that connectivity restrictions caused the banking compromises or enabled a specific intrusion. Its defensible effects are on visibility, coordination, validation, and trust.

Those effects create a secondary attack surface. Users seeking secure communications or ordinary services turn to VPNs, media players, and utilities from informal channels, and recent TAG-182 activity used exactly such lures to distribute MarkiRAT to Farsi-speaking users inside and outside Iran. Iranian surveillance operators can exploit attempts to bypass connectivity restrictions, while foreign intelligence services and criminals can exploit the same demand, putting journalists, NGOs, diaspora communities, and Iran-exposed employees at risk from several directions.

Our assessment is that the internal Iranian cyber environment is defined by three overlapping risks:

• attack against national and institutional infrastructure;
• concentration of critical services and recovery dependencies; and
• surveillance of people attempting to operate through the resulting disruption.

Treating only the first as “cyberwar” misses a substantial part of the operational reality.

Defender Priorities

For enterprise defenders, the central question extends beyond who is attacking now. It includes which access predates escalation, which trusted paths remain, and what an operator could do with them under different tasking. Review identity, cloud, RMM, and service-provider relationships against that standard, and map shared dependencies: which nominally separate services would fail together, and which recovery paths rely on the same identity provider, carrier, or administrator as production.

For incident responders, persona-led operations weaponize uncertainty. Publication, doxxing, and direct approaches to employees may begin before scoping is complete, so technical response, legal review, communications, and physical-safety support cannot operate sequentially.

For OT operators, the priorities remain clear: remove direct internet exposure; place remote access behind authenticated gateways with phishing-resistant MFA; restrict programming-mode and logic changes; enforce source and time restrictions on vendor access; monitor engineering workstations and industrial protocols; preserve offline project files and known-good configurations; and verify that recovery does not depend on the same identity, virtualization, or management environment that may have been compromised. Defenders do not need to wait for perfect attribution before removing a preventable route to operational disruption.

Outlook

Renewed kinetic escalation is likely to increase intelligence tasking, opportunistic targeting, public claims, and pressure to produce visible effects. It does not fundamentally change the access paths available to Iranian operators. Identity compromise, trusted administration, remote-management tooling, service providers, exposed internet-facing systems, and high-trust individuals remain the mechanisms most likely to produce results.

We assess with high confidence that the near-term base case is continued espionage and access development accompanied by persona-led coercion and a large volume of lower-impact activity. That includes credential theft, mailbox and cloud compromise, recruitment-themed social engineering, exploitation of trusted service relationships, and inflated public claims. Most of it will be operationally persistent but individually less dramatic than the public discussion of “cyberwar” implies.

We assess with moderate confidence that Iran-linked operators will attempt selective disruption where three conditions coincide: usable access already exists, the victim has political or symbolic value, and the expected effect can be achieved without an unacceptable risk of escalation or exposure. Administrative and management systems are particularly relevant because they translate ordinary enterprise access into organization-wide effects.

OT activity will remain dangerous but uneven. The most likely incidents involve exposed systems, weak credentials, poorly controlled remote engineering, or known vulnerabilities rather than sophisticated manipulation of segmented safety-critical environments. A technically simple compromise can still cause serious consequences when the target environment is fragile or excessively connected.

Based on the public record, we assess with moderate confidence that a coordinated, national-scale campaign intended to disable the U.S. power grid or multiple critical sectors simultaneously is a lower-likelihood, high-impact contingency rather than the near-term base case. There is public evidence of repeated targeting, reconnaissance, and exploitation of exposed industrial systems, along with limited operational disruption.

There is no public evidence at the time of writing of the synchronized access, specialized preparation, and cross-sector execution required to support claims of an imminent nationwide grid-down operation.

Official U.S. statements confirm that USCYBERCOM and USSPACECOM layered non-kinetic effects into the opening military campaign to disrupt Iranian communications and sensor networks, but they do not disclose the specific systems, accesses, techniques, duration, or reversibility involved. That is representative of the wider conflict: analysts may know a cyber-enabled effect occurred while lacking the evidence to attribute a particular outage or reconstruct the operation.

Outside Iran, public claims are likely to outpace independently verified effects; inside Iran, consequential effects may outpace public reporting, and attribution will often remain harder than impact assessment.

Analysts should stop treating logos and claim volume as the principal units of analysis. The more useful unit is access plus mission plus dependency, evaluated against evidence quality. Security teams should focus on the trusted pathways that let an otherwise ordinary compromise become wartime leverage.

Assessment current as of 21 July 2026.

Executive Summary

• SentinelLABS has been tracking sustained cyberespionage activity against several Pakistani law enforcement organizations, taking place from February 2024 to April 2026.
• All these actors converged on Balochistan Police over this period, bringing both a partner and an adversary of Pakistan to the same police force in a province shaped by a separatist insurgency and the regional tensions it has drawn in.
• At Balochistan Police, the compromised assets included servers hosting web applications that manage police and citizen data, such as criminal and biometric records.
• A suspected China-nexus actor planted implants in one of the web applications, which serves both police staff and citizens, weaponizing a tool of Pakistan’s police digitalization against its users.
• Pakistani law enforcement organizations attract cyber collection because they hold information on Pakistan’s internal security that regional powers have an incentive to pursue.
• For China, the likely primary concern is the safety of its nationals, the target of repeated deadly attacks Pakistan has failed to prevent, leading Beijing to assess that threat for itself rather than rely on its partner alone.
• For India, the strongest motive is probably its rivalry with Pakistan, with Balochistan Police offering insight into the security posture of a Pakistani province prominent in wider mutual accusations over cross-border support for militancy.

Overview

Suspected China- and India-nexus threat actors carried out intrusions into several Pakistani law enforcement organizations between 2024 and 2026. Our analysis of C2 netflow data revealed that suspected China- and India-nexus threat actors operating PlugX, ShadowPad, Cobalt Strike, and Remcos infrastructure have converged on this victim class.

All of these threat actors were active against Balochistan Police, the principal police force serving the Pakistani province of the same name, at various points between 2024 and 2026. The affected assets spanned network appliances and servers hosting web applications that manage biometric records, hotel and tenant registrations linked to national identity records, criminal case files, and personnel records. A suspected China-nexus threat actor also compromised one of these web applications, deploying custom implants masquerading as a portal update. The application is used by police staff and by citizens interacting with law enforcement through it, and the compromise put both user groups within the threat actor’s reach.

When multiple cyberespionage actors operate against law enforcement institutions of a single state, the convergence itself is a signal of target value. What draws them is a particular kind of institution: one that holds the government’s internal security picture, what it knows about the threats inside its borders, and how it acts against them. Each of the states suspected to be behind the activities covered in this post has its own stake in the threats monitored by Pakistani law enforcement.

Strategic Motives | Distrust and Accusations

The China-nexus activity is most likely motivated primarily by concern for the safety of Chinese nationals. Their presence across Pakistan is substantial, tied in large part to the China-Pakistan Economic Corridor (CPEC), Beijing’s flagship Belt and Road infrastructure program in the country. Chinese nationals have been the target of repeated deadly attacks, some of which were claimed by the Balochistan Liberation Army (BLA), a Baloch separatist group opposed to China’s presence in the Pakistani resource-rich southwest. Notable attacks include the October 2024 Karachi airport attack and the March 2024 suicide bombing in northwestern Pakistan.

The attacks have fueled explicit Chinese dissatisfaction with Pakistani counter-militancy performance. In October 2024, China’s Ambassador to Pakistan publicly called them “unacceptable”, warning that the security situation was the main obstacle to CPEC. The threat to Chinese nationals remains unresolved. As recently as January 2026, China’s Minister of Public Security and Pakistan’s Interior Minister agreed to expand counterterrorism coordination, deepen police training exchanges, and establish a special unit in Islamabad to protect Chinese nationals.

Pakistani law enforcement is a natural collection target for China. The data it holds would let China assess the security environment its nationals face independently, rather than relying on a partner whose protection has repeatedly fallen short.

For the India-nexus activity, which was focused on Balochistan, the strongest motive is probably the adversarial security relationship between India and Pakistan, in which the province is a recurring flashpoint. Pakistan has long accused India of backing the Baloch insurgency, describing the BLA as an “Indian proxy“, a charge it has not publicly substantiated and that India denies. India, in turn, accuses Pakistan of backing the militant groups behind attacks in Indian-administered Kashmir, which Pakistan denies.

The Baloch insurgency is a front in the antagonism between the two states, and Balochistan Police would hold the operational record of how Pakistan manages the province’s security. For India, that material could offer visibility into a conflict at the center of the accusations and counter-accusations between them.

Balochistan Police is the same law enforcement institution the China-nexus actors were active against, approached from the opposite direction. To China, it is the police force of a partner that cannot be trusted to protect Chinese nationals in Balochistan. To India, it is the police force of a rival, with deep insight into the security of a province central to the friction between the two states.

Intrusions Into Pakistani Law Enforcement Organizations

We observed the highest concentration of intrusions at Balochistan Police. They affected network appliances and web servers hosting several of its web applications, one of which, the Complaint Management System (CMS), drew particular attention. The next two sections discuss the impacted assets in greater detail.

We also identified compromised infrastructure associated with several other Pakistani law enforcement organizations:

• the Khyber Pakhtunkhwa Police, the police force of Khyber Pakhtunkhwa province;
• the Islamabad Police, which serves the Islamabad Capital Territory;
• the Punjab Safe Cities Authority (PSCA), an autonomous government body that operates the integrated command, control and communication system for the police in the major cities of Punjab province.

We group the C2 activity we observed against all these targets into four clusters, each associated with a single malware family or tool: PlugX, ShadowPad, Cobalt Strike, and Remcos.

Because we cluster based on tooling, not on actor attribution, the number of threat actors behind each C2 activity cluster differs. We associate the Remcos cluster with a single actor, while the PlugX, ShadowPad, and Cobalt Strike clusters are built on shared or commodity tooling and may each involve more than one operator.

The table below presents the constituent servers of each C2 activity cluster, along with the first and last dates on which any of its servers communicated with Pakistani law enforcement infrastructure. The figure that follows shows how each cluster’s activity was distributed across the targeted organizations over time.

The sections below cover the basis for each cluster’s attribution and its broader victimology. The observation windows we present there are generally wider, spanning all per-cluster victims.

C2 Activity Cluster | PlugX and ShadowPad

PlugX and ShadowPad point to China-nexus cyberespionage groups on the basis of the tooling itself, since both are backdoors shared among multiple such groups. The victimology we observed for PlugX (between 27 February and 28 September 2024) and ShadowPad (between 3 August and 1 December 2024) reinforces this assessment.

Beyond Pakistani law enforcement, victimology for PlugX and ShadowPad includes government, foreign affairs, defense, nongovernmental, and research entities across South, Southeast, Central, and East Asia, the Arabian Peninsula, and Southeast Europe, consistent with China-aligned collection.

C2 Activity Cluster | Remcos

We attribute the Remcos C2 server 89.31.121[.]220 to a suspected India-nexus threat actor, which Recorded Future tracks as TAG-179. Its infrastructure, tooling, and TTPs overlap to varying degrees with those of the threat actors tracked by Kaspersky as Mysterious Elephant and by Qihoo 360 as APT-C-08 (a.k.a. Bitter).

Our data shows that TAG-179 has been intensifying its activities and diversifying its TTPs since early 2025. This trend aligns with the prior research from Kaspersky and Qihoo 360, which documents in detail the tooling and infection chains used across 2025 and 2026.

Notably, Qihoo 360 describes a chain that delivers a Remcos backdoor configured with the same server that constitutes our Remcos C2 activity cluster (89.31.121[.]220). The IOC table of this report lists several lure files and backdoor components that we associate with TAG-179.

Among the lures is one with direct relevance to Pakistani law enforcement: it displays a decoy document posing as an operational plan for the repatriation of illegal foreigners, including Afghan Citizen Card (ACC) holders. These are Afghan nationals who have been granted temporary registration in Pakistan and are targeted for deportation under Pakistan’s Illegal Foreigners’ Repatriation Plan.

The decoy document outlines coordination among district-level police forces, the National Database and Registration Authority (NADRA, an agency of the Pakistani Ministry of Interior), and Pakistani intelligence organizations. Its subject matter is consistent with the Pakistani law enforcement victimology of TAG-179, making it an example of a plausible lure against this target class.

Within our observation window for the Remcos C2 activity (from 20 November 2025 to 21 April 2026), the victimology outside Pakistani law enforcement includes government, defense, foreign affairs, intelligence, research, and manufacturing entities across South and Southeast Asia, and the Middle East.

C2 Activity Cluster | Cobalt Strike

Although Cobalt Strike is a commodity tool that carries no inherent attribution, we attribute with medium confidence both servers in this C2 activity cluster to China-nexus threat actors.

The C2 traffic to 142.171.183[.]8, spanning 13 September 2024 to 5 December 2025, reveals victimology extending beyond Pakistani law enforcement to government, academic, telecommunications, and non-governmental entities across South, East, and Southeast Asia, the Middle East, and South America, in line with a China-aligned targeting profile. Among these entities are Tibetan Buddhist organizations in Taiwan, a long-standing Chinese cyberespionage interest.

For 193.42.25[.]65, we observed C2 communications only with Balochistan Police infrastructure, taking place between 7 November and 5 December 2024. 193.42.25[.]65 also served as next-stage infrastructure for one of two implants deployed on the Balochistan Police CMS web application. We trace these implants to a Chinese-speaking developer through related samples sharing the same development environment, a topic we discuss in greater detail in a later section.

Balochistan Police | Compromised Assets

Across the four C2 activity clusters, C2 communications involving the following Balochistan Police assets took place between 2 June 2024 and 9 April 2026:

• two network appliances;
• web servers hosting several Balochistan Police web applications;
• a Fortinet FortiMail appliance that had served as Balochistan Police’s primary inbound email gateway.

At the time of this activity, the FortiMail appliance was no longer the designated inbound email gateway, but it remained operational on the network and may have continued to process outbound or internal mail relay traffic. Its compromise may therefore have additionally exposed email traffic it processed.

Many of the web applications hosted on the affected servers are part of the Smart Police Station initiative, an EU-supported effort to modernize Balochistan policing and improve how it serves the public through digitalization. Throughout the threat actor activities, the web servers hosted a mix of public-facing applications through which citizens and businesses access policing services, alongside restricted police applications protected by firewalls against unauthorized external access.

The table below summarizes the application functions as described in publicly available documentation.

If the threat actors could reach the data stores backing these applications from the compromised servers, the data they could obtain would span police personnel records, criminal case files, biometric records, stolen vehicle records, hotel guest check-in records, tenant registration records, and citizen complaints. Together, it would provide broad visibility into Balochistan Police’s operational posture, capabilities, and intelligence activities.

Balochistan Police | CMS Compromise

The CMS web application, accessible at cms.balochistanpolice[.]gov[.]pk and hosted on one of the affected Balochistan Police web servers, was also compromised. Based on shared infrastructure and a common focus on Balochistan Police, we associate this intrusion with the threat actor operating 193.42.25[.]65, a constituent of the Cobalt Strike C2 activity cluster.

The landing page at cms.balochistanpolice[.]gov[.]pk features a login interface and a separate search form.

Access behind the login interface is highly likely restricted to law enforcement personnel. Stolen login credentials for the portal, which we retrieved from information stealer logs published on the dark web, reveal a consistent naming convention across the recovered usernames: a ps- prefix (most probably denoting “police station”) followed by a district or city within Balochistan, such as ps-barkhan.

In contrast, the search form, which posts to /Complaint/PublicSearch and accepts a complaint reference number and mobile number, is evidently intended to allow citizens to check the status of a filed complaint.

We therefore assess with high confidence that the CMS application serves two distinct user groups: law enforcement personnel and citizens.

Based on VirusTotal data, two variants of an implant named cms_plugin.exe were uploaded to cms.balochistanpolice[.]gov[.]pk/client%20scripts/ in late 2024, one written in the Rust programming language, the other compiled as a .NET executable. The Rust executable is a malware stager that downloads a payload from 193.42.25[.]65 and executes it. We could not retrieve the next stage at the time of analysis.

The .NET executable masquerades as 360Safe.exe, a component of the endpoint security software 360 Safe Guard from the Chinese vendor Qihoo 360. It reflectively loads an assembly implementing an AsyncRAT client, which is configured to use 41.216.188[.]140 as its C2 server. The assembly has a PDB path of D:\codedome\case\six\Client\Client2\obj\Debug\Client2.pdb.

Pivoting on the D:\codedome prefix, we identified multiple additional samples highly likely built in the same development environment. Several are AsyncRAT clients that share implementation patterns with the one embedded in cms_plugin.exe, such as variable naming and string obfuscation, reinforcing a common origin beyond the shared PDB prefix. Some contain Chinese-language terms in pinyin in their PDB paths, such as xinshi (likely 新式, “new type” or “new variant”), and one includes log messages in simplified Chinese. These indicators point to a Chinese-speaking developer behind the samples linked by the D:\codedome prefix, including the one deployed on the Balochistan Police CMS application.

Pivoting on the cms_plugin.exe filename, we identified a third malware stager functionally similar to the Rust variant, also downloading the next stage from 193.42.25[.]65.

Both cms_plugin.exe samples downloading from 193.42.25[.]65 display the message Update Complete! Please refresh the page upon execution, mimicking an update for the CMS portal.

The fake update prompt, combined with the cms_plugin.exe filename and the hosting location in the portal’s /client scripts/ directory, indicates that the implants were targeted at users of the CMS platform: police staff, citizens checking complaint status, or both. Successful infection would grant the threat actor access to the victim’s device. In the case of police personnel, this could provide a foothold into internal police networks and access to operational data beyond the CMS platform. In the case of citizens, it would enable surveillance of those who have filed complaints through the platform.

Conclusion

The intrusions we cover in this post show how domestic security institutions can become high-value intelligence targets when the threats they monitor overlap with foreign intelligence requirements. Balochistan Police sits at such an intersection, attracting cyberespionage activity from both a partner and an adversary of Pakistan. For China-nexus actors, access to its systems could support independent assessment of threats to Chinese nationals and interests in the country. For India-nexus actors, such access provides visibility into how Pakistan manages security in a province central to its adversarial relationship with India.

The compromise of the Complaint Management System web application adds a second dimension to the activity against Balochistan Police, extending the threat actor’s reach beyond the initially compromised environment. By hosting implants in a portal used by both citizens and law enforcement personnel, the threat actor turned a tool built to make policing in Pakistan more accessible and accountable to the public into a malware delivery mechanism. This weaponization widened the collection surface from the application and its data to the users interacting with it.

The multi-actor convergence on Balochistan Police points to a structural consequence of digital policing. Systems built to centralize records, workflows, and public interaction can also centralize intelligence value by bringing together operational, institutional, and civilian data across connected environments. Law enforcement infrastructure in that setting is no longer just the digital backbone of policing but intelligence terrain, and it will be treated as such by any adversary who can reach it.

Indicators of Compromise

SHA-1 Hashes

IP Addresses

URLs

Executive Summary

• Compaction is a context-management pattern used across agent systems to compress prior context into a denser working state for long-running tasks.
• SentinelLABS evaluated OpenAI’s native Responses API implementation against our automated malware analysis evaluation harness to measure real-world impact on task quality and cost.
• Compaction reduced input tokens by ~86% with no measurable change to the aggregate evaluation score.
• Our analysis found that compaction can significantly reduce the cost and noise of long-running security workflows without sacrificing task quality.

OpenAI introduced native compaction in a March 2026 engineering post describing extensions to the Responses API. However, the underlying idea is not unique to OpenAI. Anthropic, Google, and other agent frameworks such as LangChain all expose or document related approaches under different names.

The core problem these systems address is familiar to anyone who has built an agentic system: context accumulates faster than it stays relevant, and eventually the model is carrying more history than signal. At that point, task quality degrades and costs climb without a corresponding improvement in output.

OpenAI’s solution was to build compaction directly into the runtime so developers would not need to build custom summarization and state-carrying systems themselves. The company noted that compaction is the mechanism Codex relies on for long-running coding tasks, which positions it as load-bearing infrastructure rather than a convenience feature.

At SentinelLABS, we set out to evaluate how well OpenAI’s compaction would work for automated binary analysis, a domain with its own particular demands on agent memory and state management.

Why Malware Analysis Is a Hard Problem for Agents

Our evaluation harness gives a model access to a decompiler and asks it to complete the following:

• Identify important functions and follow code paths
• Interpret strings, APIs, call relationships, and data structures
• Rename functions or variables based on observed behavior
• Propose types or object models and explain what the malware is doing

We compare the model’s output against golden reference analysis and written reports across scoring metrics for correctness and completeness. To achieve a high score, the model needs to maintain a working theory for the slice of the binary it is analyzing, track evidence already collected, and hold open questions alongside provisional conclusions.

Malware analysis is an iterative process with a low-reward signal. A human analyst might inspect one function, learn something, pivot to another function, revise their theory, check a data structure, then return to update their original conclusion. Models do well in our evaluation where execution paths have straightforward continuity. They struggle when connections are unclear or require multiple rounds of investigation.

In observing model performance, we noticed that the agent tended to carry an increasing volume of tokens between tasks. The pattern is familiar to anyone who has run a ReAct-style agent on a non-trivial problem. Each turn adds more context until the model is dragging the full history of the run behind it, most of which stopped being useful several steps ago.

A human analyst working the same problem does not keep every raw observation equally active. They compress state between sessions. They remember that a function is probably the command dispatcher, that a particular object looks like transport state, that a given path was a dead end. They also write findings in a notebook, externalizing what they want to persist so they do not have to hold it all in working memory.

That distinction between working memory and durable memory is where compaction becomes architecturally useful.

How We Applied Compaction

Our system uses compaction to carry forward the working state: the current goal, what has already been tried, what was learned, which hypotheses remain active, what evidence changed the plan, and what questions are still open.

Specific findings and exact artifacts live outside the model context in durable storage. For malware analysis this includes logs and tool outputs, decompiled functions, intermediate artifacts, and ground-truth comparisons. When the agent needs exact evidence, it retrieves it from storage rather than relying on the compacted context to preserve it verbatim. In our use case tool use and response for binary exploration created increasingly large prompts. As the model used more tools to explore the space it added new findings – not all of them necessary. We leveraged compaction to summarize those tool calls and findings into more manageable chunks to maintain the working memory of the agent, but dramatically reduce the operational token overhead.

This split is what makes compaction measurable. A workflow that relies on compaction to preserve exact evidence will eventually produce incorrect answers when summarization or compression flattens crucial details. A workflow where compaction handles working memory and durable storage handles facts can be evaluated cleanly, because the boundary between the two is explicit.

Results

Across several long-running malware analysis agent evaluations we, compared runs with compaction enabled against runs without it.

The token reductions were substantial. The aggregate evaluation score holding flat is what matters. We were able to carry forward enough state for the workflow to continue correctly while dramatically reducing the context processed per run.

One metric did decrease: domain object modeling, meaning the model’s ability to recover the higher-level objects and structures that explain the malware’s behavior. This is not a minor caveat. For malware analysis, object and type recovery is often where the most analytically valuable conclusions are drawn.

Our read is that compaction occasionally flattened structural reasoning that would have been useful later, and it reinforces why exact artifacts must live in durable storage rather than the compacted context.

Nevertheless, our research found that compaction made longer-running analysis practical and preserved the main evaluation outcome while doing it.

Implementation

Model providers expose compaction capabilities differently. For example, Anthropic and OpenAI both provide server-side compaction; however, OpenAI exposes an additional standalone compaction endpoint. This allows developers to solve the same problem at different points in their workflows as explained below.

Server-side Compaction

This is the simpler starting point. The Responses API call includes a compaction threshold in context_management. When context length crosses that threshold, the API compacts prior context automatically during the response, with no separate call required from the application.

response = client.responses.create(
    model="gpt-5.5",
    input=conversation,
    store=False,
    context_management=[
        {"type": "compaction", "compact_threshold": 200000}
    ],
)

Standalone Compaction

This gives explicit control over when compaction happens. The application sends a context window to /responses/compact and receives a compacted context window back, which then becomes the input for the next response call.

compacted = client.responses.compact(
    model="gpt-5.5",
    input=long_input_items,
)
next_input = [
    *compacted.output,
    {
        "type": "message",
        "role": "user",
        "content": next_user_message,
    },
]
response = client.responses.create(
    model="gpt-5.5",
    input=next_input,
    store=False,
)

For our malware analysis workflows, standalone compaction was useful at phase boundaries. For example, compacting after initial triage before entering deeper function analysis. This also lets you inspect metrics before and after compaction, which is useful for identifying where specific evidence is being compressed and whether that compression affects downstream scoring.

The important constraint with the standalone endpoint is to treat the returned compacted window as the next canonical context window. Do not prune it manually unless the workflow has a specific and well-understood reason to do so.

A practical decision rule for choosing between them:

How to Use Compaction

The main takeaway for us was that compaction works best when it is part of a broader context-engineering strategy.

1. Separate working memory from source-of-truth artifacts. Compaction is appropriate for the immediate state the model needs to continue working. Exact evidence belongs somewhere else. This boundary matters both for correctness and for being able to evaluate whether the compacted run behaved correctly.
2. Compact long-running workflows. Compaction has the most impact when a task involves many steps and repeated tool use. Short interactions have little to compress.
3. Start with server-side compaction. For most agent loops it is the fastest way to learn whether compaction helps. Move to standalone when compaction policy becomes part of the task or evaluation design.
4. Do not evaluate on cost alone. A run can become significantly cheaper while losing task quality, depending on what was compacted. Resource and outcome metrics need to be tracked together.
5. Preserve negative information. Long-running agents need to remember what failed, not just what worked. Failed paths carry state that informs subsequent decisions, and compaction can discard them if the workflow does not explicitly mark them as worth preserving.
6. Treat compaction as lossy until proven otherwise. Use evaluations, traces, and artifact comparisons to verify that the compacted run still behaves correctly. Our domain object modeling result is a reminder that what looks like clean compression can still affect specific downstream capabilities.

Conclusion

Compaction is part of a broader shift from prompt engineering to context engineering. Prompt engineering concerns what we ask of the model in a single turn whereas context engineering concerns what the model gets to see across multiple turns: what gets compressed, what gets retrieved, what gets written to durable state, and what gets discarded.

For agents running long-horizon tasks, context engineering may be as important as model selection. A strong model with poor state management will lose the thread on a complex task. A model with better context discipline may make steadier progress and complete more tasks. That tradeoff compounds quickly across the kind of multi-step security workflows we are trying to evaluate.

Without compaction, realistic long-running security agent workflows become too large, noisy, and expensive to measure cleanly. With it, the scope of what is practical to evaluate expands. We view compaction not just as a tool for making agents cheaper, but as part of the infrastructure required to evaluate whether they actually work.

References

• OpenAI: Responses API + Codex https://openai.com/index/equip-responses-api-computer-environment
• Anthropic: Long-Running Agent Harnesses https://www.anthropic.com/engineering/effective-harnesses-for-long-running-agents
• LangChain: Deep Agents + Context Compression https://docs.langchain.com/oss/python/deepagents/context-engineering
• Jason Liu: Compaction As Something To Evaluate https://jxnl.co/writing/2025/08/30/context-engineering-compaction
• Context Compression For Long-Horizon Agents https://arxiv.org/abs/2601.07190

Executive Summary

• SentinelLABS has analyzed a Rust macOS implant that embeds a 3.5 KB prompt-injection payload of 38 fabricated “system” messages, built to steer an LLM-assisted triage pipeline into aborting or refusing its analysis.
• Command-and-control runs over a Telegram Bot API polling loop, with AES-GCM payloads over certificate-pinned TLS.
• The implant self-redacts its Telegram bot token in its own runtime output, denying it to anyone who captures logs or crash artifacts.
• We assess with high confidence that the implant, which we track as macOS.Gaslight, belongs to a cluster of DPRK-aligned macOS activity.

Introduction

In early June, an Apple XProtect update surfaced a Mach-O sample that had been uploaded to VirusTotal on May 22. The XProtect rule targets the file purely on its hash rather than on any internal strings or bytecode, yet the sample remains undetected by static engines on VirusTotal at the time of writing. The binary is ad hoc signed and carries the identifier endpoint-macos-aarch64-5555494492fc075f441637fb9d894913dde3a2ea.

The sample is a macOS implant and infostealer written in Rust. Its most notable feature is an embedded cascade of fabricated system-failure messages, designed to make an LLM-assisted triage agent doubt its own session. It attacks the agent’s perception, rather than the sandbox it runs in. Accordingly, we dub this family macOS.Gaslight.

We assess with high confidence that this implant sits within a cluster of DPRK-aligned macOS activity. Apple’s XProtect detects the sample under the rule MACOS_BONZAI_COBUCH, and SentinelLABS associates the BONZAI signature family with North Korean threat activity. A sibling BONZAI sample is additionally caught by Apple’s AIRPIPE rule, a family SentinelLABS likewise ties to North Korean activity.

Command & Control | Telegram Bot API

The implant’s command-and-control channel is a Telegram Bot API getUpdates polling loop. The polling branch executes only when no webhook is registered, and the dispatch handler keys on three Telegram error codes: BotBlocked, InvalidToken, and Conflict.

Telegram issues a Conflict response when two instances of the same bot token poll simultaneously, so the implant treats that response as an implicit single-instance lock. A second copy detects the conflict and terminates.

Once the bot token validates and the polling loop is active, the operator can task the implant, including through the interactive shell described below, and collected data is returned over the same channel using Telegram’s multipart attach:// file-upload mechanism.

The bot token, the chat ID (tg_room_id), and the rest of the operator configuration are supplied at runtime and are absent from this sample. Accordingly, the analysis below is based on static examination of the binary and its embedded payloads.

Transport Hardening | AES-GCM Over Pinned TLS

All C2 payloads are encrypted with AES-GCM, implemented using the pure-Rust aes-gcm 0.10.3 crate, with a fresh nonce generated per message via CCRandomGenerateBytes. The AES key is supplied at runtime through the aes_key field in the operator config rather than being embedded in the sample.

On top of the payload encryption, the implant configures a custom certificate trust anchor and calls SecTrustSetAnchorCertificatesOnly, restricting TLS trust evaluation to that anchor alone. This certificate pinning rejects connections intercepted by a standard proxy CA, frustrating network-level inspection of the operator’s traffic.

The implant also honors the host’s proxy settings, reading the active system proxy configuration via SCDynamicStoreCopyProxies and routing the traffic from its Rust reqwest/hyper networking stack accordingly. As a result, the C2 can still reach the operator on networks that force outbound connections through a proxy.

Taken together, those choices make the channel harder to inspect in transit while still allowing it to operate in tightly managed enterprise networks.

Operator Access | An Interactive Shell

After validation and activation, the operator gains an interactive shell. Two co-located command menus define six verbs.

There is some evidence of a seventh command, focus, but we were unable to recover further details from our analysis.

The implant creates an IOPMAssertionCreateWithName power-management assertion to prevent system sleep. Blocking sleep sustains long-running C2 polling and collection across periods of user inactivity, making the implant resilient to a host that would otherwise idle.

All told, the functionality provides the operator with a persistent, interactive foothold on the host.

The 15-Field Cross-Platform Operator Config

The implant reads its operator configuration using serde, a widely used Rust serialization and deserialization framework.

The operator provides the implant with a config blob at runtime and serde fills in a predefined set of fields. By default, serde matches incoming config keys to fields by their literal names, so the entire configuration schema of 15 field names is baked into the binary as plaintext.

tg_room_id           	
github_token         	
github_repo          	
github_polling_interval 
main_upload_url      	
main_base_url        	
aes_key              	
payload_path_linux   	
payload_path_macos   	
persist_name_linux   	
persist_name_macos   	
persist_type_linux   	
persist_type_macos   	
init_python_enable   	
persist_enable       	

The Linux- and GitHub-related fields are not exercised in the sample, suggesting the schema is an operator-facing interface to a broader toolset.

Collection | A Gated Python Stealer With Its Own Runtime Supply Chain

The implant carries a 6.6 KB base64-encoded Python script which serves as a data collection module. Once decoded, it harvests:

• Chrome, Brave, Firefox, and Safari browser data
• Terminal command histories
• Installed application listings
• A running-process snapshot via ps aux
• System hardware and software profile via system_profiler
• A raw copy of login.keychain-db

Collected artifacts are archived to temp/collected_data.zip and uploaded to the operator via Telegram.

SentinelLABS has previously documented Atomic macOS Stealer (AMOS) harvesting the same login keychain copy and browser data and an early Rust macOS stealer targeting login.keychain-db in 2023.

A separate 2 KB base64-encoded bash installer fetches and stages a self-contained cpython-3.10.18 interpreter from the astral-sh/python-build-standalone project. The installer, a prerequisite for deploying the Python stealer, carries the literal constants PY_VERSION=3.10.18 and BUILD_DATE=20250708 and targets both arm64 and x86_64 macOS. The widespread use of emojis and strict adherence to comment headers are consistent with LLM-generated output.

Microsoft has previously described macOS stealers bundling Python via PyInstaller and Nuitka. However, fetching a standalone CPython build from astral-sh/python-build-standalone at runtime has not been previously documented as far as we are aware. The separation keeps the main implant in Rust while letting the operator stage a fuller Python-based collection environment only when needed.

We identified init_python_enable in the serde schema as the configuration field associated with both the stealer and installer. Consistent with our earlier observations, we found no exact runtime branch logic, so we describe both only as configurable capabilities present in the binary.

Persistence | An Apple System-Service Masquerade

Persistence is achieved through a LaunchAgent. This implant’s plist carries the Label value com.apple.system.services.activity. Masquerading within Apple’s com.apple.* namespace is a tactic widely used in many macOS malware families, including those previously tied to DPRK-linked activities.

In order to write a valid absolute path to itself into the plist’s ProgramArguments array, the implant resolves its own executable location at runtime via __NSGetExecutablePath.

The implant’s persistence behavior is controlled through the persist_enable serde config field, and again we did not recover a separate static branch that would confirm exactly how installation is triggered in this sample.

OPSEC | Bot-Token Self-Redaction

Telegram bot tokens are a known weak point in bot-based C2. If the token can be recovered, defenders can use it as a detection artifact and even query the Telegram Bot API directly, exposing the bot’s chat history, operator commands, and registered webhooks. macOS.Gaslight addresses this with a self-redaction routine built into its Telegram URL constructor.

When the URL path segment is the 4-byte literal “file” (0x656c6966 little-endian), the constructor substitutes the token that follows with the hardcoded placeholder file/token:redacted, preventing the live bot credential from appearing in any diagnostic output or error string the implant produces at runtime.

The logic prevents anyone who captures the process’s logs, errors, or crash artifacts from determining the bot token, which otherwise is only available in the config itself and cannot be recovered from the sample.

NVISO Labs has previously noted that most documented Telegram bot abuse embeds recoverable tokens; macOS.Gaslight’s runtime self-redaction appears novel relative to that reporting.

A Prompt Injection That Targets the Analyst

The implant does little conventional anti-analysis. It resolves its API calls at runtime through dlsym so as to avoid embedding them in the static symbol table, and it locates its own executable dynamically rather than from a hardcoded path.

What makes the sample notable is its attempt to mislead the analyst reading the output. It carries a 3.5 KB Markdown-fenced blob of hostile data containing 38 fabricated “system” messages delimited with {{DATA}} tokens.

The {{DATA}} tokens and the surrounding Markdown fence mimic an LLM triage harness’s own prompt scaffold, blurring the boundary between untrusted sample data and trusted instructions.

The scaffold contains fake system messages about token expiry, out-of-memory kills, disk exhaustion, and repeated operation failures. It also plants bogus warnings about injection vulnerabilities and static-analysis flags. The aim is to push an LLM agent into aborting, truncating, or refusing analysis.

Check Point first documented this kind of analyst-targeting prompt injection publicly in 2025, describing a Windows proof-of-concept that used a single direct-instruction prompt injection to evade AI-based detection.

Socket has since documented a Hades supply-chain payload whose stealer opens with a fake prompt-injection header to pollute AI-assisted analysis, while the leaked Shai-Hulud code carried an “Anthropic Magic String” intended to stop Claude Code from analyzing it. Each relied on a single injected block or header rather than the 38-message harness-spoofing cascade seen here.

Previous SentinelLABS research, by contrast, examined malware that uses LLMs to generate or support capability at runtime rather than interfere with analyst tooling.

Conclusion

macOS.Gaslight packs considerable capability into a single, persistent Rust binary, bundling a credential and session-data stealer, an interactive shell, and a self-staged Python collection chain behind a hardened Telegram C2. Aside from the runtime-fetched standalone CPython interpreter, these are all established macOS tradecraft.

However, macOS.Gaslight is noteworthy for its analyst-targeting prompt injection, an attempt to weaponize the LLM-assisted triage pipelines that increasingly sit in the reverse-engineering loop.

Anyone building such tooling should treat the contents of the samples they triage as adversarial input, never as instructions, and be prepared to keep hostile content out of the model entirely. As LLM-assisted analysis becomes routine, defenders should expect more samples built to exploit it.

Indicators of Compromise

In this final video in our LABScon Replay series from LABScon 25, we present the keynote from SentinelLABS’ own Juan Andrés Guerrero-Saade (JAGS), VP, Intelligence & Security Research and Senior Technical Fellow.

In this engaging talk, Juan argues that cybersecurity is reaching the end of its experimental era. Years of piling complexity onto non-standardized software stacks have produced systems that are difficult to steer and costly to manage through human attention alone.

A key theme in the talk is JAGS’ argument that large language models change that equation. He describes them as a new source of cheap, effectively unlimited evaluative power and as a “lossy compression of human knowledge”. Used well, that kind of mechanized intelligence can give defenders a scalable way to assess, prioritize, and act without leaning so heavily on scarce human expertise. In practical terms, it lowers the cost of analysis and changes how defensive work can be done at scale.

That argument also shapes JAGS’ broader point about how security should evolve once this capability is built in rather than bolted on. Drawing on ideas from cybernetics, JAGS urges the industry to move beyond purely adversarial ‘agonistic’ design and toward systems in which human expertise and artificial evaluative power work together to produce better outcomes.

Rather than defending old product categories or familiar workflows, the talk points toward a more standardized, more automated, and more sustainable future for the industry. For anyone interested in how AI may reshape the practice, cost, and structure of cybersecurity itself, this keynote is essential viewing.

About the Author

Juan Andrés Guerrero-Saade (JAGS) is VP for Intelligence and Security Research and Senior Technical Fellow for AI Innovation, overseeing intelligence production and AI applications towards security problems. He’s also Distinguished Resident Fellow for Threat Intelligence at the Johns Hopkins SAIS Alperovitch Institute. His research work is the subject of two permanent exhibits at the International Spy Museum in Washington, DC. He’s a member of OpenAI’s Frontier Risk Council, founded the premier threat intelligence conference LABScon, and is a co-host of the Three Buddy Problem podcast.

LABScon 2026 | Call For Papers

Submission Deadline: June 19, 2026

LABScon is a unique venue for original research to be shared among peers. The benefit of an invite-only audience of researchers is that there’s no need for long preambles or introductions – speakers are encouraged to dive right into their technical findings.

• Original content only.
• Talks are 20 minutes long + 5 minutes for Q&A.
• Workshops are 90 minutes long.
• LABScon is primarily a threat intelligence and vulnerability research conference but we keep an open-mind.

About LABScon

This presentation was featured live at LABScon 2025, an immersive 3-day conference bringing together the world’s top cybersecurity minds, hosted by SentinelOne’s research arm, SentinelLABS.

Keep up with all the latest on LABScon here.

In this LABScon 25 presentation, ESET researchers Matthieu Faou and Zoltán Rusnák present the first technical evidence that Gamaredon actively facilitated Turla’s access to high-value Ukrainian targets in Ukraine.

Across incidents observed between February and June 2025, Gamaredon tooling, including PteroGraphin and PteroOdd, was used to deploy Turla’s Kazuar backdoor and, in at least one case, restore Turla’s access after the group appeared to have lost its foothold.

The talk opens with a current view of Gamaredon’s tradecraft. Still one of the most active espionage actors targeting Ukraine, the group relies on relentless spearphishing, lightweight custom tooling, and fast operational tempo to compromise military and government organizations. Matthieu and Zoltán show how those patterns continue to evolve while remaining highly effective in a wartime environment.

The researchers’ provide evidence of direct operational collaboration between Gamaredon and Turla, detailing concrete cases in which Gamaredon activity enabled Turla operations on already compromised systems. The talk offers a rare look at how Russian cyberespionage operations may divide labor in practice, with one actor establishing or maintaining access and another deploying a more advanced espionage platform to exploit it.

The talk also examines Kazuar v2 and v3, Turla’s flagship backdoor, and unpacks what those versions reveal about the group’s operational priorities. From deployment chains to capability depth, the analysis helps defenders connect initial access activity with downstream post-compromise objectives and better understand how sophisticated implants are sustained inside contested networks.

This talk is essential viewing for defenders, threat hunters, and intelligence teams tracking Russian state-aligned activity in Ukraine, particularly those interested in access brokering, inter-group collaboration, and the continuing evolution of Turla’s malware stack.

About the Authors

Matthieu Faou is a senior malware researcher at ESET where he specializes in researching targeted attacks. His main duties include threat hunting and reverse engineering of APTs. He has spoken at multiple conferences including Black Hat USA, BlueHat, Botconf, CYBERWARCON, NorthSec, and Virus Bulletin.

Zoltán Rusnák is a senior malware researcher at ESET, with a decade of experience in malware analysis and research. He has worked extensively on identifying and systematically monitoring major botnet families, including the infamous Emotet and Trickbot. His background in large-scale botnet tracking has been central to his current research on Gamaredon.

LABScon 2026 | Call For Papers

Submission Deadline: June 19, 2026

LABScon is a unique venue for original research to be shared among peers. The benefit of an invite-only audience of researchers is that there’s no need for long preambles or introductions – speakers are encouraged to dive right into their technical findings.

• Original content only.
• Talks are 20 minutes long + 5 minutes for Q&A.
• Workshops are 90 minutes long.
• LABScon is primarily a threat intelligence and vulnerability research conference but we keep an open-mind.

About LABScon

This presentation was featured live at LABScon 2025, an immersive 3-day conference bringing together the world’s top cybersecurity minds, hosted by SentinelOne’s research arm, SentinelLABS.

Keep up with all the latest on LABScon here.

When a company suffers a cyber breach, its stock price often takes a hit, but the timing, depth, and duration of that reaction are far less predictable. In this LABScon25 presentation, Mick Baccio and Scott Roberts explore whether public indicators of breach activity can be used to anticipate market response before formal disclosure.

Drawing on sources such as EDGAR filings, executive blog posts, and social media chatter, the speakers examine how public breadcrumbs can reveal incident activity early enough to support an opportunistic trading strategy. At the center of the talk is their “15/30” hypothesis: short the stock after a breach becomes visible, then flip long as the market recovers.

To test the idea, Baccio and Roberts used AI-assisted data collection to build a dataset of public disclosures relating to “material” cyber breaches at U.S. companies. They then compared their initial, intuition-led model with a more structured time-series analysis based on a Hidden Markov Model to see whether a more rigorous timeline could improve performance.

Along the way, the presentation digs into real-world breach cases, market misreads, and missed opportunities. One particularly useful comparison looks at two similarly sized casino operators hit by ransomware around the same period, illustrating how market outcomes can diverge sharply depending on factors such as response strategy, disclosure dynamics, and investor perception.

After working through a set of highly mixed results, the speakers arrive at what they call “quantitized nihilism”, a conclusion that questions many of the assumptions analysts bring to cyber-event trading and how the market actually values cyber failures.

This talk is essential viewing for security practitioners, investors, and analysts interested in the messy intersection of cyber risk, public disclosure, and market psychology.

About the Authors

Mick Baccio is a globally recognized security strategist with a career spanning offensive operations, threat intelligence, and national-level incident response. He currently advises organizations around the world through his role at Splunk, helping security leaders improve operations through data-informed approaches. Mick was the first Chief Information Security Officer for a U.S. presidential campaign (2020), and previously served in the Obama White House as the Chief of the Threat Intelligence Branch.

Scott J. Roberts is a cybersecurity leader with over 15 years of experience specializing in cyber threat intelligence and threat hunting after leadership roles at GitHub, Apple, and Splunk. He blends machine learning with traditional intelligence frameworks to track and disrupt nation state and criminal adversaries.

LABScon 2026 | Call For Papers

Submission Deadline: June 19, 2026

LABScon is a unique venue for original research to be shared among peers. The benefit of an invite-only audience of researchers is that there’s no need for long preambles or introductions – speakers are encouraged to dive right into their technical findings.

• Original content only.
• Talks are 20 minutes long + 5 minutes for Q&A.
• Workshops are 90 minutes long.
• LABScon is primarily a threat intelligence and vulnerability research conference but we keep an open-mind.

About LABScon

This presentation was featured live at LABScon 2025, an immersive 3-day conference bringing together the world’s top cybersecurity minds, hosted by SentinelOne’s research arm, SentinelLABS.

Keep up with all the latest on LABScon here.

Executive Summary

• SentinelLABS has identified PCPJack, a credential theft framework that worms across exposed cloud infrastructure and removes artifacts associated with TeamPCP,  a threat actor persona who claimed several high-profile supply chain intrusions throughout early 2026.
• The toolset harvests credentials from cloud, container, developer, productivity, and financial services, then exfiltrates the data through attacker-controlled infrastructure while attempting to spread to additional hosts.
• PCPJack targets exposed services including Docker, Kubernetes, Redis, MongoDB, RayML, and vulnerable web applications, enabling both external propagation and lateral movement inside victim environments.
• Unlike typical cloud-focused malware, PCPJack does not deploy cryptominers; the services it targets suggest monetization through credential theft, fraud, spam, extortion, or resale of stolen access.

Overview

On 28 April 2026, SentinelLABS located a script through a Kubernetes-focused VirusTotal hunting rule that stood out from known cloud hacktools: the script’s first actions are to evict and delete tools associated with the TeamPCP attack group, leading us to call the toolset PCPJack. Analyzing this script led us to discover a full framework dedicated to cloud credential harvesting and propagating onto other systems, both internal and external to the victim’s environment.

TeamPCP stood out in early 2026 following the group’s February compromise of Aqua Security’s Trivy vulnerability scanner. The incident enabled several downstream attacks, including the compromise of LiteLLM, an open-source library that routes requests across widely used LLM providers. TeamPCP also announced a partnership with the VECT ransomware group to monetize the data stolen through their cloud environment attacks.

Many of the services targeted by the PCPJack framework are similar to the early TeamPCP/PCPCat campaigns from December 2025, before the high-visibility campaigns of early 2026 brought significant attention to TeamPCP and purportedly led to changes in group membership. We believe this could be a former operator who is deeply familiar with the group’s tooling.

The types of credentials collected by the framework suggest PCPJack’s targeting motivations are primarily to conduct spam campaigns and financial fraud, or to simply monetize stolen credentials to actors with these focuses. The inclusion of enterprise productivity software like Slack and business database services expands the focus to extortion attacks. Notably, neither of the two toolsets we identified from the attacker’s staging server performed any cryptocurrency mining, a stark departure from typical multi-disciplinary cloud attack campaigns.

First Toolset | bootstrap.sh & Python Worms

The infection begins with bootstrap.sh, a shell script designed for Linux systems. This script serves only to set up the environment and download additional payloads. bootstrap.sh sets several key variables, including PAYLOAD_HOST, which is set to hxxps://spm-cdn-assets-dist-2026[.]s3[.]us-east-2[.]amazonaws[.]com, a legitimate Amazon Simple Storage Service (S3) resource that was likely registered by the attacker for unauthorized purposes.

The main functionality of bootstrap.sh is:

1. Create /var/lib/.spm/ working directory
2. Check public IP against operator’s blocklist: this prevents the attacker from infecting their own infrastructure
3. Find and remove processes or artifacts that match naming conventions referencing TeamPCP or PCPcat process list, services, paths, or containers
4. Install Python 3.6+ via available package manager: apk, apt, dnf, pacman, yum or zypper
5. Create a Python virtual environment and install requests, cryptography, and pyarrow
6. Download six Python modules from the attacker’s S3 URL in the following order: worm.py, parser.py, lateral.py, crypto_util.py, cloud_ranges.py, cloud_scan.py
7. Rename modules to their on-disk names (see the list of downloaded payloads below)
8. Establish persistence:
   1. If run as root: create sys-monitor.service, which runs monitor.py, aka worm.py, an orchestrator script
   2. If not root, create two crontabs: one runs every 5 minutes to check if monitor.py is running, the other starts monitor.py if it is not running
9. Launch monitor.py
10. Self delete using rm -f "$0"

The following table itemises the downloaded payloads:

The logic targeting TeamPCP files stands out: each of the artifacts has been associated with TeamPCP in public reporting, though BORING_SYSTEM is mentioned only sparsely. We initially considered that this toolset could be a researcher removing TeamPCP’s infections. However, analysis of the later-stage payloads indicates otherwise. When exfiltrating system information and credentials, the PCPJack operator even collects success metrics on whether TeamPCP has been evicted from targeted environments in a “PCP replaced” field sent to the C2.

Infection Flow

The infection begins with bootstrap.sh, which executes the orchestrator script, monitor.py (aka worm.py). The orchestrator imports a set of purpose-built modules for credential parsing (utils.py), lateral movement (_lat.py), C2 message encryption (_cu.py), cloud IP range lookups (_cr.py), and cloud scanning (_csc.py).

Rather than let the modules find their own dependencies, the orchestrator injects them at runtime with shared references, ensuring all components operate with the same credential and movement handles without hardcoding inter-module imports.

The scanning module, _csc.py, receives the lateral movement engine, the cloud range lookup function, and the credential parser all via injection from the worm. This design keeps each module independently minimal while the orchestrator alone holds the full dependency graph, making the framework harder to analyze in isolation. No single imported file reveals the complete picture without visibility into monitor.py.

Sensitive strings are stored in the source code as a hex-encoded blob instead of clear text. When a script runs, it obtains the actual value by calling function _d(), which is near the top of each Python module, against the encoded hex string containing the sensitive content.

The function decrypts it by XORing each byte against the MD5 hash of the string urllib3.poolmanager, a name chosen to look like a reference to a common Python web library. PCPJack’s author encrypted the constants that would immediately identify the malware’s infrastructure. Despite this, the actor failed to encrypt the Telegram bot token in bootstrap.sh and the credential decryption key in crypto_util.py, so the operational security awareness only goes so far.

monitor.py | Orchestrator Script

The monitor.py script, which was hosted on the attacker’s staging server as worm.py and had persistence established by bootstrap.sh, is the main script driving the toolset. The script starts with logic designed to make the script appear like a benign system monitoring utility that collects metrics about the system.

While this is valuable information for the attacker, we believe it is an attempt to help the script blend in if spotted by an administrator, given that the posted information also includes data about the types of systems being targeted by the toolset.

Local Credential Theft

On each compromised host, monitor.py executes a shell pipeline that steals:

• .env files and config files
• Environment variables filtered for secrets, API keys, DB & SMTP creds
• SSH private keys and targets from known_hosts, ~/.ssh/config, and bash history
• AWS IMDS credentials
• Kubernetes service account tokens
• Docker secrets (/run/secrets/)
• Cryptocurrency wallets (wallet.dat files, Ethereum keystores, Solana keys)

A separate scan walks the local /etc, /home, /opt, /root, /srv, /var/lib , and /var/www directories looking for config or secret files, and _mgr searches through git history for deleted secrets. The results are parsed by utils.py.

Target Selection & Propagation

After credential extraction, the command checks for prior installation (/var/lib/.spm/worm.py or /var/lib/.spm/monitor.py) and, if clean, downloads and executes bootstrap.sh from the C2 payload host.

Propagation targets come from parquet files that the worm downloads directly from Common Crawl, a legitimate web scan archival nonprofit with a rich history of furnishing AI models with vast amounts of training data harvested from the web. The URL is extracted from obfuscated variables _CI and _CB.

The tool picks parquet files containing url_host_name columns, and iterates through those hostnames. Each monitor.py node gets a window of parquet files based on the date or a seed index (SPM_SEED_IDX), which gives the attacker distributed coverage without central coordination. A deduplication set, variable _sh, is stored in memory to prevent re-scanning. This list is capped at 15 million entries.

This module spreads the toolset to targets by exploiting several vulnerabilities in web technologies, including the ubiquitous React2Shell flaw:

Command & Control

The framework uses Telegram for C2. An infected system posts data to one channel and checks another to receive commands from a pinned message. Most of the commands are self-explanatory. RUN downloads a module from the attacker’s payload storage, saves it as run_script.py, and executes the script. The PARQUET command gives the node a new index to parse from the parquet file, meaning the operator can manually override previously chosen attack ranges.

utils.py | Credential Extractor

This script handles credential extraction using regular expressions to identify and categorize stolen keys and secrets. The logic centers on a wide variety of online services, many of which pertain to bulk messaging services, cryptocurrency/FinTech, cloud or web application services.

Finance & Enterprise

SMTP & Bulk Messaging Services

Web & Cloud Services

Interestingly, the actor’s regular expression matching includes credentials for FTX, a crypto exchange that went bankrupt in a high-profile case in 2022. This suggests the actor adapted the matching logic from an older tool, or that it was inserted erroneously through LLM code generation.

lateral.py | Internal Network Lateral Movement

The lateral.py or _lat.py script performs reconnaissance on the infected system and the assets it connects to, enabling internal propagation. The script runs only once and writes a lateral_done file to the working directory; if that file is found, the script exits. This is likely to improve stealth and reduce the likelihood of network security alerts.

The Kubernetes spreading logic _lk checks for a Kubernetes service account token, which is present inside pods mounted in a cluster, then uses the service account to authenticate with the Kubernetes management API to enumerate namespaces and pods in the cluster. The script runs commands against each container to:

• Extract credentials from a list of file names and paths associated with secret stores
• Harvest SSH private keys
• Query the AWS Instance Metadata Service (IMDS); this works only in environments where IMDSv2 is not strictly enforced and goes against modern default configurations and best practices

_lk also reads Kubernetes Secrets and ConfigMaps directly via the API, base64-decodes their values, which works even when pod execution is denied by role-based access controls (RBAC). Lastly, it attempts a container escape by mounting the host filesystem to a new container, enabling the attacker’s tools to interact with the host system.

The Docker propagation function _ld checks for the local Docker socket at /var/run/docker.sock, then scans the network for services running on ports 2375 or 2376. When found, the script connects to the Docker API through the management daemon, lists all running containers, and executes the same credential harvesting script as seen in the Kubernetes routine. If connected to a remote host, the spreader will bind-mount the root filesystem of the machine running the Docker management service to the remote instance’s /host path, which creates a container escape.

When Redis is found, _rec dumps the configuration, then calls the Redis KEYS command to scan database key names for secrets, passwords, tokens, and API keys, and GETs their values. For persistence, _rwc performs a Redis cron rewrite, resulting in a cron job that fires bootstrap.sh every 5 minutes as root.

lateral.py targets several other services running within the victim’s environment:

• RayML Clusters: scans port 8265, submits a Python job via the API to extract credentials and download bootstrap.sh
• MongoDB: scans port 27017, enumerates databases & extracts credentials

The SSH propagation module _ls searches SSH key store locations on the infected machine and parses  ~/.ssh/known_hosts, ~/.ssh/config, and .bash_history for username and host combinations. It then pulls SSH keys from harvest.jsonl, a file containing credentials found by other lateral movement techniques earlier. These combinations are tried against any hosts running SSH. On access, it runs bootstrap.sh on the remote machine to propagate the worm.

crypto_util.py | Data Encryption

PCPJack’s framework uses the crypto_util.py (aka _cu.py, imported as a module named _crypto) script to encrypt credentials. It is called by monitor.py to exfiltrate the encrypted data before it is sent to the attacker’s Telegram channel.

The encrypt_message function:

• Generates an X25519 keypair for each message chunk
• Performs ECDH against a hardcoded attacker public key set to variable _RPK = "6d4imqQ/s/GfQCVcybdcjfTe/PMYHtZN8ZGHnEXSbRo="
• Uses the raw shared secret directly as a ChaCha20-Poly1305 key to encrypt the data
• Splits output into 2800-byte chunks: the __main__ test block validates against Telegram’s 4096-character message limit
• Packs each encrypted chunk by concatenating the ephemeral public key (32 bytes), a random nonce (12 bytes), and the ciphertext, then base64-encodes the result and prepends a emoji

If the cryptography library is not installed, the function silently falls back to sending plaintext, meaning credentials may be exfiltrated unencrypted during some infections.

The decrypt_message function requires the private key corresponding to variable _RPK. The test keypair in __main__ (PRIVATE_KEY) is a matching test pair. If a researcher could access the attacker’s Telegram channel, there is a reasonable chance they could decrypt the stolen credentials sent to the channel. During our testing, the Telegram API responded that the bot token was invalid, although the malware was actively hosted and being distributed during this time.

cloud_ranges.py | Cloud Service Provider IPs

The cloud_ranges.py (aka _cr.py) module is relatively small and simple: it collects a list of IP addresses assigned to AWS, Azure, Cloudflare, Cloudfront, Fastly, and Google Cloud Platform (GCP). The approach is to query URLs from each provider that host information about the cloud service IP ranges, which change periodically.

This allows the attacker to avoid hardcoding IPs into the script which may be outdated. Once the information is retrieved, the cloud ranges are written to a file at /var/lib/.spm/_cr/ranges.json. The data is refreshed every 24 hours.

cloud_scan.py | External Propagation

The final module is cloud_scan.py (aka _csc.py), which scans external cloud services and attempts to propagate by looking for ports indicating exposed Docker, Kubernetes, MongoDB, RayML, or Redis services.

When a target responds on a matching port, cloud_scan.py scans the entire /24 subnet for the responding IP and runs infection logic imported from lateral.py.

For Docker, Redis, and RayML targets, this includes installing persistence via bootstrap.sh. Docker is targeted through a privileged container with host escape, Redis through cron injection, and RayML through a weaponized job submission.

Kubernetes and MongoDB targeting results only in credential harvesting. cloud_scan.py queries unauthenticated Kubernetes API endpoints to dump secrets and it scrapes MongoDB collections for credentials, but does not establish persistence.

Infrastructure

bootstrap.sh contains a hardcoded list of attacker infrastructure IPs excluded from targeting. Perhaps a nostalgic nod to the presumed retired cloud attack group TeamTNT, each of these IPs are VPS servers geolocated to Germany. Given the complex dynamics that could drive this attacker to focus on killing processes associated with TeamPCP activity, it is reasonable to scrutinize whether these IPs actually belong to the attacker behind PCPJack.

• 38.242.204[.]245
• 38.242.237[.]196
• 38.242.245[.]147
• 83.171.249[.]231
• 161.97.129[.]25
• 161.97.135[.]154
• 161.97.163[.]87
• 161.97.186[.]175
• 161.97.187[.]42
• 193.187.129[.]143
• 213.136.80[.]73

The IPs are relatively minimal in their online footprint, but the available data suggests management infrastructure and potentially different malicious activity. 38.242.245[.]147 has hosted lastpass-login-help[.]com, clearly a phishing domain to harvest LastPass master credentials: a motive that aligns with this toolsets heavy credential harvesting focus.

Second Toolset | Credential Harvester & Sliver Beacons

We also identified another toolset on the attacker’s payload delivery server unrelated to the previous one. The file check.sh is an 858-line shell script that handles everything before the beacon phones home. The script detects CPU architecture and pulls the matching Sliver binary: update.bin, update-386.bin, or update-arm.bin, depending on the system architecture.

The binary is saved locally as /var/tmp/apt-daily-upgrade to blend in with system processes. Simultaneously, check.sh sweeps IMDS endpoints, Kubernetes service accounts, Docker instances, and /proc/*/environ for credentials from 30+ services, many through a dropped Python script called  extractor.py.

Targets include many services covered by the bootsrap.sh framework, with several standout new additions: Anthropic, Digital Ocean, Discord, Google API, Grafana Cloud, HashiCorp Vault, OnePassword, and OpenAI keys.

The script then exfiltrates stolen data to hxxps://cdn[.]cloudfront-js[.]com:8443/u, a typosquatted domain mimicking CloudFront, over ports 443 or 8443. It finishes by SSH-spraying up to 10 lateral targets before self-deleting.

Sliver ELF Binaries

The update binaries are Sliver C2 beacons compiled with the garble obfuscation tool, which scrambles Go type names and removes build metadata to hinder signature-based detection. Despite the obfuscation, several indicators remained across the analyzed binaries: protobuf field tags (name=BeaconID), interface method names (GetC2URI, GetBeaconInterval), and multiple Sliver-specific RPC strings including PivotListener, PivotPeerEnvelope, WGSocksServer, and WGTCPForwarder among others.

The binaries form a deployment set: update.bin, the 64-bit variant, targets modern Intel-based cloud infrastructure and includes CPU feature detection for Intel Sapphire Rapids, a powerful processor present in many cloud environments.

update-386.bin is a 32-bit variant that serves as a capability-identical fallback for legacy servers or 32-bit containers.

update-arm.bin is designed for ARM processors. Interestingly, each of the binaries have different garble seeds, meaning they were compiled separately and hinders conclusive attribution to the same developer.

Conclusion

Overall, the two toolsets are well developed and indicate that the owner values making code as a modular framework, despite some redundancies in behavior. The occasional operational security lapses were interesting, particularly their choice to encrypt everything except for Telegram credentials and their own alleged infrastructure.

In the threat actor ecosystem, there is constant churn and turnover between groups: something TeamPCP alluded to before their main social media account was suspended.

We have no evidence to suggest whether this toolset represents someone associated with the group or familiar with their activities. However, the first toolset’s focus on disabling and replacing TeamPCP’s services implies a direct focus on the threat actor’s activities rather than pure cloud attack opportunism. There are plenty of other cloud credential harvesting campaigns which have other forensic artifacts that could be considered when performing a pre-installation cleanup early during an intrusion.

Compared to similar cloud threat actors, PCPJack stands out for its complete lack of cryptominers in all tooling we analyzed. Nearly all moderately-sophisticated cloud threat campaigns deploy XMRig or similar at some point, including several of TeamPCP’s campaigns. This campaign does not, and it deliberately removes the miner functions associated with TeamPCP. Desite that, this actor has well-defined scopes for extracting cryptocurrency credentials.

Mitigations and Recommendations

The impacts of PCPJack and similar toolsets range from data exposure and extortion to financial impacts of an attacker with access to high-limit, enterprise API services.

Organizations can defend against these threats by adhering to cloud and web application security best practices. Credential management will mitigate the majority of these credential harvesting techniques: use an enterprise-wide vault or secret management service and ensure access to those stores is never stored to a file saved in clear text.

Ensure that authentication mechanisms follow industry standards: require MFA from service accounts rather than an API key alone. In AWS environments, ensure that IMDSV2 is enforced across all services to prevent credential theft and consider allow-listing downloads only from approved S3 resources.

Even when systems are not exposed to the internet, ensure authentication is required to manage services like Docker and Kubernetes, as these are popular lateral movement targets which can enable much deeper access through connected nodes, and restrict scopes on Kubernetes service accounts to adhere to the principle of least privilege.

Indicators of Compromise

Domains

IP Addresses
The following IP addresses are hardcoded into bootstrap.sh and labelled as attacker infrastructure:

File Hashes | SHA-1

File System

HTTP Request Indicators

Strings

In this LABScon 25 presentation, Joe FitzPatrick explores how networked devices manufactured overseas have quietly become indispensable to everything from small-business prototyping labs to roadside infrastructure. He argues that the safeguards meant to manage the risks these devices introduce are, in practice, largely ineffective.

Starting with recent reports of undocumented cellular radios found in solar inverters used in U.S. highway infrastructure, Joe notes that adding that kind of connectivity to a device with an exposed serial port takes minutes and can be done by anyone: the manufacturer, the installer, or someone who came along later.

From there he covers the familiar mechanisms by which banned hardware finds its way into supply chains anyway, through relabeling and FCC-certified modular components, before turning to mandatory product activation in consumer devices like drones and 3D printers, and what it actually takes to use them without phoning home.

The deeper problem is that small businesses and infrastructure operators are genuinely dependent on imported hardware because it works and it’s affordable. A significant amount of it runs on devices that connect to foreign entities by default, and there’s no clean domestic alternative.

Joe concludes that import bans don’t fix problems that exist equally in domestic products, and that trade policy is the wrong tool for what is fundamentally a consumer safety problem. His preferred alternatives are right to repair with offline use guarantees, hardware and firmware bills of materials, and comprehensive privacy legislation.

This talk is essential viewing for security practitioners concerned about hardware supply chain risks, the unexpected connectivity of critical infrastructure, or the US’s deep dependence on foreign-manufactured consumer electronics.

About the Author

Joe FitzPatrick (@securelyfitz) is an Instructor and Researcher at SecuringHardware.com. Joe has spent most of his career working on low-level silicon debug, security validation, and penetration testing of CPUs, SoCs, and microcontrollers. He has spent the past decade developing and delivering hardware security related tools and training, instructing hundreds of security researchers, pen testers, and hardware validators worldwide. When not teaching Applied Physical Attacks training, Joe is busy developing new course content or working on contributions to the NSA Playset and other misdirected hardware projects, which he regularly presents at all sorts of fun conferences.

LABScon 2026 | Call For Papers

Submission Deadline: June 19, 2026

LABScon is a unique venue for original research to be shared among peers. The benefit of an invite-only audience of researchers is that there’s no need for long preambles or introductions – speakers are encouraged to dive right into their technical findings.

• Original content only.
• Talks are 20 minutes long + 5 minutes for Q&A.
• Workshops are 90 minutes long.
• LABScon is primarily a threat intelligence and vulnerability research conference but we keep an open-mind.

About LABScon

This presentation was featured live at LABScon 2025, an immersive 3-day conference bringing together the world’s top cybersecurity minds, hosted by SentinelOne’s research arm, SentinelLABS.

Keep up with all the latest on LABScon here.

Update | 07 May 2026

Executive Summary

• SentinelLABS has uncovered a previously undocumented cyber sabotage framework whose core components date back to 2005, tracked as fast16.
• fast16.sys selectively targets high-precision calculation software, patching code in memory to tamper with results. By combining this payload with self-propagation mechanisms, the attackers aim to produce equivalent inaccurate calculations across an entire facility.
• This 2005 attack is a harbinger for sabotage operations targeting ultra expensive high-precision computing workloads of national importance like advanced physics, cryptographic, and nuclear research workloads.
• fast16 predates Stuxnet by at least five years, and stands as the first operation of its kind. The use of an embedded customized Lua virtual machine predates the earliest Flame samples by three years.
• The name ‘fast16’ is referenced in the infamous Shadow Brokers’ leak of NSA’s ‘Territorial Dispute’ components. An evasion signature instructs operators: “fast16 *** Nothing to see here – carry on ***”

Overview

Our investigation into fast16 starts with an architectural hunch. A certain tier of apex threat actors has consistently relied on embedded scripting engines as a means of modularity. Flame, Animal Farm’s Bunny, ‘PlexingEagle’, Flame 2.0, and Project Sauron each built platforms around the extensibility and modularity of an embedded Lua VM. We wanted to determine whether that development style arose from a shared source, so we set out to trace the earliest sophisticated use of an embedded Lua engine in Windows malware.

Lua is a lightweight scripting language with a native proficiency for extending C/C++ functionality. Given the appeal of C++ for reliable high-end malware frameworks, this capability is indispensable to avoid having to recompile entire implant components to add functionality to already infected machines. We did not find an indication of direct shared provenance, but our investigation did uncover the oldest instance of this modern attack architecture.

Lua leaves a distinctive fingerprint. Compiled bytecode containers start with the magic bytes 1B 4C 75 61 (\x1bLua), followed by a version byte, and the engine typically exposes a characteristic C API and environment variables such as LUA_PATH. Hunting for these traits across mid-2000s malware collections surfaced a sample that initially looked unremarkable: svcmgmt.exe.

svcmgmt.exe | A 2005 Lua-Powered Service Binary

On the surface, svcmgmt.exe appears to be a generic console‑mode service wrapper from the Windows 2000/XP era.

A closer look reveals an embedded Lua 5.0 virtual machine and an encrypted bytecode container unpacked by the service entry point.

The developers extended the Lua environment to include:

• a wstring module for native unicode handling
• a built‑in symmetric cipher, exposed through a function commonly labelled b, used to decrypt embedded data
• multiple modules that bind directly into Windows NT filesystem, registry, service control, and network APIs.

Even by itself, svcmgmt.exe already looks like an early high-end implant, a modular service binary that hands most of its logic to encrypted Lua bytecode. The binary includes a crucial detail: a PDB path that links the binary to the kernel driver fast16.sys.

fast16 | A Nagging Mystery from The Shadow Brokers Leak

Buried in the binary’s strings is a PDB reference:

C:\buildy\driver\fd\i386\fast16.pdb

At first glance, the path is structured like any other compiler artifact: an internal build directory, a component name (fast16), and an architecture hint (i386). However, in this case there’s a mismatch. The string appears inside of a service-mode executable, and yet the driver\fd\i386\fast16 segment of the pdb string clearly refers to a kernel driver project.

Following that clue led us to a second binary, fast16.sys:

This kernel driver is a boot-start filesystem component that intercepts and modifies executable code as it’s read from disk. Although a driver of this age will not run on Windows 7 or later, for its time fast16.sys was a cut above commodity rootkits thanks to its position in the storage stack, control over filesystem I/O, and rule-based code patching functionality.

In April 2017, almost 12 years after the compilation timestamp, the same filename, “fast16” appeared in The Shadow Brokers leak. Dr. Boldizsár Bencsáth’s research into Territorial Dispute points to a text file, drv_list.txt. The 250KB file is a short list of driver names used to mark potential implants cyber operators might encounter on a target box as “friendly” or to “pull back” in order to avoid clashes with competing nation-state hacking operations.

The guidance for one particular driver, ‘fast16’, stands out as both unique and particularly unusual.

The string inside svcmgmt.exe provided the key forensic link in this investigation. The pdb path connects the 2017 leak of deconfliction signatures used by NSA operators with a multi-modal Lua‑powered ‘carrier’ module compiled in 2005, and ultimately its stealthy payload: a kernel driver designed for precision sabotage.

svcmgmt.exe | Architecture of the Carrier

The core component of fast16, svcmgmt.exe, functions as a highly adaptable carrier module, changing its operational mode based on command-line arguments.

• No arguments: Runs as a Windows service.
• -p: Sets InstallFlag = 1 and runs as a service (Propagate/Install & Run).
• -i: Sets InstallFlag = 1 and executes Lua code (Install & Execute Lua).
• -r: Executes Lua code without setting the install flag (Execute Lua).
• Any other argument (<filename>): Interprets as a filename, and spawns two children: the original command and one with the -r argument (Wrapper/Proxy Mode).

Internally, svcmgmt.exe stores three distinct payloads, including encrypted Lua bytecode that handles configuration, its propagation and coordination logic, auxiliary ConnotifyDLL, and the fast16.sys kernel driver.

By separating a relatively stable execution wrapper from encrypted, task-specific payloads, the developers created a reusable, compartmentalized framework that they could adapt to different target environments and operational objectives while leaving the outer carrier binary largely unchanged across campaigns.

The Wormlets and Early Evasion Architecture

The early 2000s saw a large number of network worms. Most were written by enthusiasts, spread quickly, and carried little or no meaningful payload. fast16 originates from the same period but follows a completely different pattern indicative of its provenance as state-level tooling. It’s the first recorded Lua-based network worm, and was built with a highly specific mission.

The carrier was designed to act like cluster munition in software form, able to carry multiple wormable payloads, referred to internally as ‘wormlets’. The svcmgmt.exe module performs the following steps:

1. Prepares the configuration, defining the payload path, service details, and target IP ranges.
2. Converts the configuration values to wide-character strings for the C layer.
3. Escalates privileges and installs the carrier executable as the SvcMgmt service, then starts it.
4. Optionally, based on the configuration setting, deploy the kernel driver implant fast16.sys.
5. Releases the wormlets. In this particular configuration, only one wormlet slot is populated with an SCM wormlet that looks for network servers, copies the payload over a network share and starts that remote service.
6. Repeats the process indefinitely, sleeping for the configured initial delay between waves, until a failure threshold or external kill condition is reached.

The wormlets were stored in the carrier’s internal storage:

The single deployed wormlet found in svcmgmt.exe (the SCM wormlet) exemplifies a simple but effective propagation strategy based on native Windows capabilities and weak network security. It targets Windows 2000/XP environments and relies on default or weak administrative passwords on file shares. All spreading is done through standard Windows service-control and file-sharing APIs, an early example of propagation that leans on built-in administration features rather than custom network protocols.

Before this workflow runs, a pre-installation kill-switch checks the environment. The ok_to_install() routine calls ok_to_propagate() and propagation is only allowed if it’s manually forced or if it’s made sure common security products aren’t found by checking for associated registry keys. The routine walks a list of vendor keys and aborts installation if any of them are present, preventing deployment into monitored environments.

For tooling of this age, that level of environmental awareness is notable. While the list of products may not seem comprehensive, it likely reflects the products the operators expected to be present in their target networks whose detection technology would threaten the stealthiness of a covert operation:

HKLM\SOFTWARE\Symantec\InstalledApps
HKLM\SOFTWARE\Sygate Technologies, Inc.\Sygate Personal Firewall
HKLM\SOFTWARE\TrendMicro\PFW
HKLM\SOFTWARE\Zone Labs\TrueVector
HKLM\SOFTWARE\F-Secure
HKLM\SOFTWARE\Network Ice\BlackIce
HKLM\SOFTWARE\McAfee.com\Personal Firewall
HKLM\SOFTWARE\ComputerAssociates\eTrust EZ Armor
HKLM\SOFTWARE\RedCannon\Fireball
HKLM\SOFTWARE\Kerio\Personal Firewall 4
HKLM\SOFTWARE\KasperskyLab\InstalledProducts\Kaspersky Anti-Hacker
HKLM\SOFTWARE\Tiny Software\Tiny Firewall
HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Uninstall\Look n Stop 2.05p2
HKCU\SOFTWARE\Soft4Ever
HKLM\SOFTWARE\Norman Data Defense Systems
HKLM\SOFTWARE\Agnitum\Outpost Firewall
HKLM\SOFTWARE\Panda Software\Firewall
HKLM\SOFTWARE\InfoTeCS\TermiNET

A separate user-mode component, svcmgmt.dll, provides a minimal reporting channel. Contained within the carrier’s internal storage, this DLL is registered through the Windows AddConnectNotify() API so that it’s called each time the system establishes a new network connection using the Remote Access Service (RAS), responsible for dial-up connections and early VPNs in the 2000s.

When invoked, the DLL decodes an obfuscated string to obtain the named pipe \\.\pipe\p577, attempts to connect to the local pipe, and writes the remote and local connection names to the pipe before closing it. The module doesn’t run independently and must be registered by a host process.

fast16.sys | A Filesystem Driver for Precision Sabotage

The kernel driver fast16.sys is the most potent component of the framework.

The driver is configured with Start=0 (boot) and Type=2 (filesystem driver) in the SCSI class group. It loads automatically at an early stage, alongside disk device drivers, and inserts itself above each filesystem device (NTFS, FAT, MRxSMB). On entry it:

• disables the Windows Prefetcher by setting the EnablePrefetcher value to 0 under the Session Manager’s PrefetchParameters key, forcing subsequent code‑page requests through the full filesystem stack,
• resolves kernel APIs dynamically using a simple XOR‑based string cipher and a scan of ntoskrnl.exe, and
• exposes \Device\fast16 and \??\fast16 with a custom DeviceType value 0xA57C, which serves as a secondary forensic marker.

The driver registers with IoRegisterFsRegistrationChange so it can attach a worker device object on top of every active and newly created filesystem device. All relevant I/O Request Packets, including IRP_MJ_CREATE, IRP_MJ_READ, IRP_MJ_CLOSE, IRP_MJ_QUERY_INFORMATION, IRP_MJ_FILE_SYSTEM_CONTROL, and associated Fast I/O paths, are routed through these worker devices.

Despite loading at boot, the kernel‑level code injection engine is only activated after the system opens explorer.exe. This design defers expensive monitoring and patching until the desktop environment is available and avoids unnecessary impact on core boot performance.

Narrow Targeting via Intel Compiler Artefacts

Once activated, fast16.sys focuses on executable files. A file is a valid target if it meets two criteria:

1. The filename ends with .EXE.
2. Immediately after the last PE section header, there is a printable ASCII string starting with Intel.

This selection logic points to executables compiled with the Intel C/C++ compiler, which often placed compiler metadata in that region. It indicates that the developers knew their target software was built with this toolchain.

For files meeting these criteria, the driver performs a PE header modification in memory. It injects two additional sections, .xdata and .pdata, and fills them with bytes from the original code section, increasing the section count and keeping a clean copy of the code. The intent is likely to increase stability while still allowing extensive patching, although without identifying the original target binaries this remains an informed hypothesis.

Rule‑Driven Patching and Floating‑Point Corruption

The patching engine is a minimalist, performance‑optimised, stateful scanning and modification tool. It is configured with a set of 101 rules, each containing pattern matching and replacement logic. To maintain performance, the engine:

• uses a 256‑byte dispatch array and only flags the starting byte values of a small number of unique patterns,
• allows wildcards inside patterns so a single rule can match several compiler‑optimised variants of the same code, and
• supports state flags that some rules can set or check, enabling multi‑stage modification sequences similar to those used by advanced antivirus scanning engines.

Most patched patterns correspond to standard x86 code used for hijacking or influencing execution flow. One injected block is different. It’s a larger and complex sequence of Floating Point Unit instructions dedicated to precision arithmetic and scaling values in internal arrays. This code is a standalone mathematical calculation function unrelated to code flow hijacking or any other typical malicious code injection.

To understand what the driver expected to see, we converted the patching rules into hexadecimal YARA signatures and ran them against a large, period‑appropriate corpus. The results showed a very low hit rate: fewer than ten files matched two or more patterns. Those matches, however, shared a clear theme. They were precision calculation tools in specialised domains such as civil engineering, physics and physical process simulations.

The FPU patch in fast16.sys was written to corrupt these routines in a controlled way, producing alternative outputs. This moves fast16 out of the realm of generic espionage tooling and into the category of strategic sabotage. By introducing small but systematic errors into physical‑world calculations, the framework could undermine or slow scientific research programs, degrade engineered systems over time or even contribute to catastrophic damage.

A sabotage operation of this kind would be foiled by verifying calculations on a separate system. In an environment where multiple systems shared the same network and security posture, the wormable carrier would deploy the malicious driver module to those systems as well, reducing the chance that an independent calculation would diverge from the corrupted output.

At this time, we’ve been unable to identify all of the target binaries in order to understand the nature of the intended sabotage. We welcome the contributions of the larger infosec research community and have included YARA rules to hunt for these patterns in the appendix below.

The Data Patching Engine

Even after deep analysis, fast16’s driver looks deceptively simple. Beneath that minimal code is a rule-driven in-memory engine that quietly patches executable code as files are read from disk.

The engine relies on a compact set of just over a hundred pattern-matching rules and a small dispatch table so it only inspects bytes that are likely to matter. Most patterns correspond to ordinary x86 instructions, but one stands out: a larger block of floating-point (FPU) code dedicated to precision arithmetic. This injected routine scales values in three internal arrays passed into the function, subtly changing calculations.

Without knowing the exact binaries and workloads being patched, we can’t fully resolve what those arrays represent, only that the goal is to tamper with numerical results, not unauthorized access, malware propagation or other common malware objectives.

The Patch Targets

Our best clues about the intended victims come from matching these patterns against large, era-appropriate software corpora. The strongest overlaps point to three high-precision engineering and simulation suites from the mid-2000s: LS-DYNA 970, PKPM, and the MOHID hydrodynamic modeling platform, all used for scenarios like crash testing, structural analysis, and environmental modeling.

LS-DYNA in particular has been cited in public reporting on Iran’s suspected violations of Section T of the JCPOA, in studies of computer modeling relevant to nuclear weapons development.

Compiler Footprints and Lineage

As we sought to understand the lineage of this unusual set of components, we noticed a quirk. Strings of the form @(#)par.h $Revision: 1.3 $ inside the binaries point to an unusual source‑control convention. The @(#) prefix is characteristic of early Unix Source Code Control System (SCCS) or Revision Control System (RCS) tooling from the 1970s and 1980s. These markers do not affect execution and are redundant in modern Windows kernel drivers.

Finding SCCS/RCS artefacts in mid‑2000s Windows code is rare. It strongly suggests that the authors of this framework were not typical Windows‑only developers. Instead, they appear to have been long‑term engineers whose culture and toolchain came from older, high‑security Unix environments, often associated with government or military‑grade work. This detail supports the view that fast16 came from a well‑resourced, long‑running development program.

A Digital Fossil with Modern Implications

svcmgmt.exe was uploaded to VirusTotal nearly a decade ago. It still receives almost no detections: one engine classifies it as generally malicious, and even that with limited confidence. For a stealthy self-propagating carrier that deploys one of the most sophisticated sabotage drivers of its era, that detection record is notable.

Together with its appearance in The Shadow Brokers ‘Territorial Dispute’ (TeDi) signatures, fast16 forces a re‑evaluation of our historical understanding of the timeline of development for serious covert cyber sabotage operations. The code shows that:

• state‑grade cybersabotage against physical targets was fully developed and deployed by the mid‑2000s,
• embedded scripting engines, narrow compiler‑based targeting and kernel‑level patching formed a coherent architecture well ahead of better‑known families, and
• some of the most important offensive capabilities in the ecosystem may still sit in collections as ‘old but interesting’ samples lacking the context to highlight their true significance.

Internally, the operation leaves very little in the way of branding. One of the few human‑readable labels is wry and understated:

*** Nothing to see here – carry on ***

For many years there were no public write-ups, no named campaign and no headline incident linked to this framework.

In the broader picture of APT evolution, fast16 bridges the gap between early, largely invisible development programs and later, more widely documented Lua‑ and LuaJIT‑based toolkits. It is a reference point for understanding how advanced actors think about long‑term implants, sabotage, and a state’s ability to reshape the physical world through software. fast16 was the silent harbinger of a new form of statecraft, successful in its covertness until today.

Acknowledgements

SentinelLABS would like to thank Silas Cutler and Costin Raiu for their contributions along the way. We dedicate this research to the memory of Sergey Mineev, APT hunter extraordinaire, who pioneered many of the techniques that enabled this discovery.

Update | 07 May 2026

We’ve updated this post to improve executable detection precision and tighten the formatting. Thanks to everyone who shared ideas along the way, and special thanks to the Broadcom Threat Hunter team for their early engagement and valuable feedback.

Appendix: Patching Engine Patterns and Target Candidates

Extracted Match Patterns

7C 02 89 C6 89 35 ?? ?? ?? ?? 89 B4 24 D0
0F 8F A5 00 00 00 A1 ?? ?? ?? ?? 83 F8 14 7D 0D
39 2D ?? ?? ?? ?? 0F 84 F4 00 00 00 8B 35 ?? ?? ?? ?? 2B 35
8B 4D 10 C1 E2 04 8B 19 83 EA 30 8B CB 49
8B 45 44 6B 00 04 D9 05 ?? ?? ?? ?? D8 B0
E9 7E 04 00 00 8B 74 24 1C 8B 54 24 14 85
83 39 63 0F 85 21 03 00 00 8B EE 85 F6 0F
75 2C 89 35 ?? ?? ?? ?? 89 05 ?? ?? ?? ?? 89 15
89 55 F4 8B F9 8B D3 03 FB C1 E2 02 89 35
DF E0 F6 C4 41 A1 ?? ?? ?? ?? 74 5A
FF 35 ?? ?? ?? ?? E8 ?? ?? ?? ?? 9D D9 E0 D9 1D ?? ?? ?? ?? 8B 4C
6A 46 68 ?? ?? ?? ?? E8 ?? ?? ?? ?? 6A 03
D8 05 ?? ?? ?? ?? D9 55 00 9C
D8 1D ?? ?? ?? ?? DF E0 F6 C4 41 B8 00 00 00 00 75 05 B8 01 00 00 00 85 C0 74 11 6A 29
0F 0F 94 C0 23 C3 33 D2
DD 05 ?? ?? ?? ?? 8B 05 ?? ?? ?? ?? 8B 15 ?? ?? ?? ?? 0F AF 05 ?? ?? ?? ?? 8B 1D ?? ?? ?? ?? 0F AF 15
68 28 00 00 00 57 E8 ?? ?? ?? ?? 8B 1D ?? ?? ?? ?? 8B 35 ?? ?? ?? ?? 0F AF 1D ?? ?? ?? ?? 8B 3D ?? ?? ?? ?? 8B 05
8B 55 88 8B 5D B0 83 7D 84 01
55 8B EC 83 EC 2C 33 D2 53 56 57 8B
48 89 84 24 9C 00 00 00 4B 0F 8F 79 FF FF FF
8B 5D 0C 8B 55 08 8B 36 8B
83 EC 04 53 E8 ?? ?? ?? ?? EB 09 83 EC 04 53
D8 E1 D9 5D FC D9 04
55 8B EC 83 EC 14 53 56 57 8B 3D ?? ?? ?? ?? 8B 0D
89 4D C8 8B FB 8B C8
8B 4C 24 0C 8B 01 83 F8 63
83 3D ?? ?? ?? ?? 00 0F 84 70 BD FF FF
BE 07 00 00 00 BF 04 00 00 00 BB 02 00 00 00
8D 1D ?? ?? ?? ?? 52 8D 05 ?? ?? ?? ?? 51 8D 15 ?? ?? ?? ?? 8D 0D ?? ?? ?? ?? 53 50 52 51 56 57 E8 ?? ?? ?? ?? 83 C4 38 EB 0E 83 EC 04
85 DB 8B 55 D4 75 2C 89 35
75 18 8D 35 ?? ?? ?? ?? 56 8D 3D
8D 1D ?? ?? ?? ?? 52 8D 05 ?? ?? ?? ?? 51 8D 15 ?? ?? ?? ?? 8D 0D ?? ?? ?? ?? 53 50 52 51 56 57 E8 ?? ?? ?? ?? EB 0E 83 EC 04 56 57 53 E8 95
D8 34 85 ?? ?? ?? ?? 8B 44 ?? ?? 8B CA
8D 04 BD ?? ?? ?? ?? 03 DF
8B EE 85 F6 0F 8E ?? ?? ?? ?? 8D 1C BD
D9 04 9D ?? ?? ?? ?? 83 ED 04 05 10 00 00 00 D8 0D
C2 08 00 A1 ?? ?? ?? ?? 8B 0C 85 ?? ?? ?? ?? 89 0E
2B DA 89 3C 03 83 3D
D9 5D C0 8B 4D C0 D9 45 E0 89 0E
8B 05 ?? ?? ?? ?? 8B 0D ?? ?? ?? ?? 0F 85 7E 00 00 00 0F AF 15
8B 55 30 8B 75 2C D8 C9 8B 45 30
8B 75 38 8B 4D 34 D8 C9 8B
55 8B EC 83 EC 2C B9 46 00 00 00 53 56 57 8B
8B 5D B0 0F 85 ?? ?? ?? ?? 8D 34 9D ?? ?? ?? ?? 8D 14 9D
B9 01 00 00 00 C1 E7 02 8B BF ?? ?? ?? ?? 8B D7 85 FF
2B FB 8B DE C1 E3 02 89 7D A0 03 5D A0 8B
D9 5D 00 D9 03 D8 0D ?? ?? ?? ?? D8 0D

Patch Target Candidate 1: LS-DYNA 970 Software Suite

The LS-DYNA suite is powerful engineering simulation software used to analyze how materials and structures behave under extreme conditions. The tool is used by engineers to simulate physical events and model conditions while avoiding expensive or dangerous experiments.

LS-DYNA is designed for handling dynamic, complex events that occur at speed, such as car crashes, explosions, impacts, metal forming, and manufacturing processes. It was commonly used by automotive companies, aerospace engineering, defense and military research, as well as manufacturing and materials science applications. LS-DYNA has been in development since 1976.

Patch Target Candidate 2: PKPM Software Suite

Practical Structural Design and Construction Software (PKPM) is a structural engineering CAD software suite widely used in China for building design. The suite comprises multiple executable modules covering the full lifecycle of structural building design, from structural layout and concrete shear design for beams and columns to seismic, wind, and load analysis for high-rise buildings.

PKPM’s core analysis engine, SATWE (Space Analysis of Tridimensional Wired Elements), handles tridimensional structural analysis across floors, beams, columns, walls, and frames. PKPM sees extensive use in Chinese civil engineering.

PKPM Concrete Code Shear Design Module

PKPM Building Structure CAD Modules

PKPM SATWE Structural Analysis Engine

Additional PKPM CAD files

Candidate 3: MOHID Software Suite

Modelo Hidrodinâmico (Portuguese for “Hydrodynamic Model” or MOHID) is an open-source water modeling system developed by MARETEC (Marine and Environmental Technology Research Center) at the Instituto Superior Técnico in Lisbon, Portugal. The software is used for marine and coastal water modeling, covering hydrodynamics, water quality simulation, sediment transport, oil spill modeling, and Lagrangian particle tracking.

At this time, we cannot definitively identify the target and welcome contributions from the broader research community to aid understanding of the intended effects of attacking this software.

Indicators of Compromise

YARA Rules

import "pe"

rule apt_fast16_carrier {
    meta:
        author = "SentinelLABS/vk"
        date = "2025-04-07"
        description = "Catches fast16 carrier, its Lua payload, and plaintext variants"
        hash = "9a10e1faa86a5d39417cae44da5adf38824dfb9a16432e34df766aa1dc9e3525"
    strings:
        $lua_magic = { 1B 4C 75 61 } //Lua bytecode magic

        //Decrypted strings
        $s1 = "build_wormlet_table"
        $s2 = "unpropagate"
        $s3 = "worm_install_failure_action"
        $s4 = "implant_install_failure_action"
        $s5 = "scm_wormlet_propagate_system"
        $s6 = "scm_wormlet_install"
        $s7 = "scm_wormlet_init"
        $s8 = "scm_copy_payload"
        $s9 = "get_logged_on_user"
        $s10 = "logged_on_program"
        $s11 = "phase_1_prop_delay"
        $s12 = "connotify_pipename"
        $s13 = "cndll_internal_name"
        $s14 = "connotify_provider_key"
        $s15 = "check_implant_reg_values"
        $s16 = "set_implant_reg_values"
        $s17 = "install_implant"
        $s18 = "implant_installed"
        $s19 = "implant_internal_name"
        $s20 = "implant_files"
        $s21 = "implant_owner"
        $s22 = "install_worm"
        $s23 = "start_worm"
        $s24 = "implant_install_failure_action"
        $s25 = "worm_install_failure_action"
        $s26 = "ok_to_propagate"
        $s27 = "no_firewall_check"
        $s28 = "scm_wormlet"
        $s29 = "implant_install_failure_action"
        $s30 = "worm_install_failure_action"

        //Encrypted strings
        $e1 = { 98 18 A1 94 24 E3 A2 4C  61 C8 AE 04 DC 4E 03 CD 0D 9D F0 }
        $e2 = { E8 76 53 6D D4 B9 6E 28  6C 5D C2 }
        $e3 = { 7D B7 14 73 F0 C0 4D 53  BB F7 0A 4A 3A 63 05 92  EC 0A 11 BC 22 59 99 05  72 05 19 }
        $e4 = { 88 5F 1B E4 45 56 75 4B  A5 3D 19 0B 3F 30 5A 85  E2 BD D0 E7 1C 13 D0 1D  BD D8 CF A1 88 DB }
        $e5 = { 88 1E 54 4E 00 C1 EF 79  AA AD 9F 50 27 B5 B8 4C  32 06 D2 7B 32 E3 AF D6  DC D2 BB 83 }
        $e6 = { 39 F9 BC E9 27 70 C4 3E  04 2A 7D E1 68 67 B7 ED  D4 41 6A }
        $e7 = { 13 FC 24 20 1F 20 74 1B  E5 5F 59 56 D7 61 3E BD }
        $e8 = { EF 94 49 63 33 41 62 F2  26 A6 48 DE 6D 7B A4 CF }
        $e9 = { 36 5F 5E E5 C1 1A 17 6A  4E B9 94 52 1B DC C6 60  CA C7 }
        $e10 = { B3 9C A3 F1 12 CC 52 74  34 5F 87 43 32 21 36 7B 2A }

        $rk1 = "HKEY_LOCAL_MACHINE\\SOFTWARE\\Symantec\\InstalledApps"
        $rk2 = "HKEY_LOCAL_MACHINE\\SOFTWARE\\Sygate Technologies, Inc.\\Sygate Personal Firewall"
        $rk3 = "HKEY_LOCAL_MACHINE\\SOFTWARE\\TrendMicro\\PFW"
        $rk4 = "HKEY_LOCAL_MACHINE\\SOFTWARE\\Zone Labs\\TrueVector"
        $rk5 = "HKEY_LOCAL_MACHINE\\SOFTWARE\\F-Secure"
        $rk6 = "HKEY_LOCAL_MACHINE\\SOFTWARE\\Network Ice\\BlackIce"
        $rk7 = "HKEY_LOCAL_MACHINE\\SOFTWARE\\McAfee.com\\Personal Firewall"
        $rk8 = "HKEY_LOCAL_MACHINE\\SOFTWARE\\ComputerAssociates\\eTrust EZ Armor"
        $rk9 = "HKEY_LOCAL_MACHINE\\SOFTWARE\\RedCannon\\Fireball"
        $rk10 = "HKEY_LOCAL_MACHINE\\SOFTWARE\\Kerio\\Personal Firewall 4"
        $rk11 = "HKEY_LOCAL_MACHINE\\SOFTWARE\\KasperskyLab\\InstalledProducts\\Kaspersky Anti-Hacker"
        $rk12 = "HKEY_LOCAL_MACHINE\\SOFTWARE\\Tiny Software\\Tiny Firewall"
        $rk13 = "HKEY_LOCAL_MACHINE\\SOFTWARE\\Microsoft\\Windows\\CurrentVersion\\Uninstall\\Look n Stop 2.05p2"
        $rk14 = "HKEY_CURRENT_USER\\SOFTWARE\\Soft4Ever"
        $rk15 = "HKEY_LOCAL_MACHINE\\SOFTWARE\\Norman Data Defense Systems"
        $rk16 = "HKEY_LOCAL_MACHINE\\SOFTWARE\\Agnitum\\Outpost Firewall"
        $rk17 = "HKEY_LOCAL_MACHINE\\SOFTWARE\\Panda Software\\Firewall"
        $rk18 = "HKEY_LOCAL_MACHINE\\SOFTWARE\\InfoTeCS\\TermiNET"

        $c1 = { 86 3A D6 02 } // A crypto constant
        $c2 = { 01 E1 F5 05 } // A crypto constant

        $code1 = { 8B 00           // mov     eax, [eax]
        2D 2F 34 21 33  // sub     eax, 3321342Fh
        } // Code to deobfuscate real storage container length

        $stor1 = { CC 00 00 00 05 00 00 00 66 69 6C 65 00 CD 00 00 00 } //Storage record with file string
    condition:
        ( uint16(0)==0x5a4d and filesize < 10MB and (
        ( 3 of ($s*) ) or
        ( 12 of ($rk*) ) or
        ( any of ($e*) ) or
        ( all of ($c*) and @c2-@c1 < 0x100 ) or
        ( $code1 ) or
        ( $stor1 )) ) or
        ( $lua_magic and 7 of ($s*) )
}

rule apt_fast16_driver {
    meta:
        author = "SentinelLABS/vk"
        last_modified = "2026-04-15"
        description = "Catches fast16 driver or related project files"
        hash = "07c69fc33271cf5a2ce03ac1fed7a3b16357aec093c5bf9ef61fbfa4348d0529"
    strings:
        $a1 = "@(#)foo.c : "
        $a2 = "@(#)par.h : "
        $a3 = "@(#)pae.h : "
        $a4 = "@(#)fao.h : "
        $a5 = "@(#)uis.h : "
        $a6 = "@(#)ree.h : "
        $a7 = "@(#)fir.h : "
        $a8 = "@(#)fir.c : "
        $a9 = "@(#)par.h : "
        $a10 = "@(#)pae.h : "
        $a11 = "@(#)fao.h : "
        $a12 = "@(#)uis.h : "
        $a13 = "@(#)ree.h : "
        $a14 = "@(#)fir.h : "
        $a15 = "@(#)myy.h : "
        $a16 = "@(#)fic.h : "
        $a17 = "@(#)ree.h : "
        $a18 = "@(#)ree.c : "
        $dev1 = "\\Device\\fast16"
        $dev2 = "\\??\\fast16"
        $pdb1 = "C:\\buildy\\"
        $pdb2 = "driver\\fd\\i386\\fast16.pdb"
        $devtype = { 68 7C A5 00 00 } // push 0A57Ch ; DeviceType
        $api1 = {50 C6 45 D4 16 C6 45 D5 2B C6 45 D6 12 C6 45 D7 3F C6 45 D8 3F C6 45 D9 3C C6 45 DA 30 C6 45 DB 32 C6 45 DC 27 C6 45 DD 36 C6 45 DE 03 C6 45 DF 3C C6 45 E0 3C C6 45 E1 3F C6 45 E2 53 } // push xored "ExAllocatePool"
        $api2 = {C6 45 A8 16 C6 45 A9 2B C6 45 AA 12 C6 45 AB 3F C6 45 AC 3F C6 45 AD 3C C6 45 AE 30 C6 45 AF 32 C6 45 B0 27 C6 45 B1 36 C6 45 B2 03 C6 45 B3 3C C6 45 B4 3C C6 45 B5 3F C6 45 B6 04 C6 45 B7 3A C6 45 B8 27 C6 45 B9 3B C6 45 BA 07 C6 45 BB 32 C6 45 BC 34 C6 45 BD 53} // push xored "ExAllocatePoolWithTag"
        $api3 = {C6 45 E4 16 C6 45 E5 2B C6 45 E6 15 C6 45 E7 21 C6 45 E8 36 C6 45 E9 36 C6 45 EA 03 C6 45 EB 3C C6 45 EC 3C C6 45 ED 3F C6 45 EE 53} // push xored "ExFreePool"
        $api4 = {C6 45 C0 16 C6 45 C1 2B C6 45 C2 15 C6 45 C3 21 C6 45 C4 36 C6 45 C5 36 C6 45 C6 03 C6 45 C7 3C C6 45 C8 3C C6 45 C9 3F C6 45 CA 04 C6 45 CB 3A C6 45 CC 27 C6 45 CD 3B C6 45 CE 07 C6 45 CF 32 C6 45 D0 34 C6 45 D1 53} // push xored "ExFreePoolWithTag"
    condition:
        filesize < 10MB and 
        ( uint16(0)==0x5a4d and
        ( ( 2 of ($pdb*) ) or
        ( $pdb1 and 1 of ($a*) ) or
        ( #devtype == 3 and
        pe.machine == pe.MACHINE_I386 and
        pe.subsystem == pe.SUBSYSTEM_NATIVE) or
        any of ($api*) or
        2 of ($dev*))) or 
        ( 6 of ($a*))
}

rule clean_fast16_patchtarget {
  meta:
    author = "SentinelLABS/vk"
    created = "2026-04-15"
    last_modified = "2026-05-07"
    description = "Detects fast16 clean patch targets. Patterns extracted directly from fast16.sys's runtime rule engine. Improved version of the rule"
    hash = "07c69fc33271cf5a2ce03ac1fed7a3b16357aec093c5bf9ef61fbfa4348d0529"

  strings:
    $el2  = { 7C 02 89 C6 89 35 ?? ?? ?? ?? 89 B4 24 D0 }
    $el3  = { 0F 8F A5 00 00 00 A1 ?? ?? ?? ?? 83 F8 14 7D 0D }
    $el16 = { 39 2D ?? ?? ?? ?? 0F 84 F4 00 00 00 8B 35 ?? ?? ?? ?? 2B 35 }
    $el26 = { 8B 4D 10 C1 E2 04 8B 19 83 EA 30 8B CB 49 }
    $el31 = { 8B 45 44 6B 00 04 D9 05 ?? ?? ?? ?? D8 B0 }
    $el32 = { E9 7E 04 00 00 8B 74 24 1C 8B 54 24 14 85 }
    $el33 = { 83 39 63 0F 85 21 03 00 00 8B EE 85 F6 0F }
    $el43 = { 75 2C 89 35 ?? ?? ?? ?? 89 05 ?? ?? ?? ?? 89 15 }
    $el45 = { 89 55 F4 8B F9 8B D3 03 FB C1 E2 02 89 35 }
    $el49 = { DF E0 F6 C4 41 A1 ?? ?? ?? ?? 74 5A }
    $el51 = { FF 35 ?? ?? ?? ?? E8 ?? ?? ?? ?? 9D D9 E0 D9 1D ?? ?? ?? ?? 8B 4C }
    $el53 = { 6A 46 68 ?? ?? ?? ?? E8 ?? ?? ?? ?? 6A 03 }
    $el56 = { D8 05 ?? ?? ?? ?? D9 55 00 9C }
    $el61 = { D8 1D ?? ?? ?? ?? DF E0 F6 C4 41 B8 00 00 00 00 75 05 B8 01 00 00 00 85 C0 74 11 6A 29 }
    $el80 = { 0F 0F 94 C0 23 C3 33 D2 }
    $el83 = { DD 05 ?? ?? ?? ?? 8B 05 ?? ?? ?? ?? 8B 15 ?? ?? ?? ?? 0F AF 05 ?? ?? ?? ?? 8B 1D ?? ?? ?? ?? 0F AF 15 }
    $el89 = { 68 28 00 00 00 57 E8 ?? ?? ?? ?? 8B 1D ?? ?? ?? ?? 8B 35 ?? ?? ?? ?? 0F AF 1D ?? ?? ?? ?? 8B 3D ?? ?? ?? ?? 8B 05 }
    $el96 = { 8B 55 88 8B 5D B0 83 7D 84 01 }
    $el97 = { 55 8B EC 83 EC 2C 33 D2 53 56 57 8B }

    $el0  = { 48 89 84 24 9C 00 00 00 4B 0F 8F 79 FF FF FF }
    $el4  = { 8B 5D 0C 8B 55 08 8B 36 8B }
    $el6  = { 83 EC 04 53 E8 ?? ?? ?? ?? EB 09 83 EC 04 53 }
    $el10 = { D8 E1 D9 5D FC D9 04 }
    $el12 = { 55 8B EC 83 EC 14 53 56 57 8B 3D ?? ?? ?? ?? 8B 0D }
    $el13 = { 89 4D C8 8B FB 8B C8 }
    $el14 = { 8B 4C 24 0C 8B 01 83 F8 63 }
    $el23 = { 83 3D ?? ?? ?? ?? 00 0F 84 70 BD FF FF }
    $el25 = { BE 07 00 00 00 BF 04 00 00 00 BB 02 00 00 00 }
    $el28 = { 8D 1D ?? ?? ?? ?? 52 8D 05 ?? ?? ?? ?? 51 8D 15 ?? ?? ?? ?? 8D 0D ?? ?? ?? ?? 53 50 52 51 56 57 E8 ?? ?? ?? ?? 83 C4 38 EB 0E 83 EC 04 }
    $el34 = { 85 DB 8B 55 D4 75 2C 89 35 }
    $el36 = { 75 18 8D 35 ?? ?? ?? ?? 56 8D 3D }
    $el37 = { 8D 1D ?? ?? ?? ?? 52 8D 05 ?? ?? ?? ?? 51 8D 15 ?? ?? ?? ?? 8D 0D ?? ?? ?? ?? 53 50 52 51 56 57 E8 ?? ?? ?? ?? EB 0E 83 EC 04 56 57 53 E8 95 }
    $el39 = { D8 34 85 ?? ?? ?? ?? 8B 44 ?? ?? 8B CA }
    $el40 = { 8D 04 BD ?? ?? ?? ?? 03 DF }
    $el41 = { 8B EE 85 F6 0F 8E ?? ?? ?? ?? 8D 1C BD }
    $el42 = { D9 04 9D ?? ?? ?? ?? 83 ED 04 05 10 00 00 00 D8 0D }
    $el59 = { C2 08 00 A1 ?? ?? ?? ?? 8B 0C 85 ?? ?? ?? ?? 89 0E }
    $el63 = { 2B DA 89 3C 03 83 3D }
    $el68 = { D9 5D C0 8B 4D C0 D9 45 E0 89 0E }
    $el70 = { 8B 05 ?? ?? ?? ?? 8B 0D ?? ?? ?? ?? 0F 85 7E 00 00 00 0F AF 15 }
    $el81 = { 8B 55 30 8B 75 2C D8 C9 8B 45 30 }
    $el94 = { 8B 75 38 8B 4D 34 D8 C9 8B }
    $el99 = { 55 8B EC 83 EC 2C B9 46 00 00 00 53 56 57 8B }

    $el30 = { 8B 5D B0 0F 85 ?? ?? ?? ?? 8D 34 9D ?? ?? ?? ?? 8D 14 9D }
    $el73 = { B9 01 00 00 00 C1 E7 02 8B BF ?? ?? ?? ?? 8B D7 85 FF }
    $el75 = { 2B FB 8B DE C1 E3 02 89 7D A0 03 5D A0 8B }

    $el46 = { D9 5D 00 D9 03 D8 0D ?? ?? ?? ?? D8 0D }

  condition:
    filesize < 200MB and uint16(0) == 0x5A4D and 2 of them
}

rule apt_fast16_patch {
	meta:
		author = "SentinelLABS/vk"
		last_modified = "2026-04-15"
		description = "Detects the fast16 patch code. May be present in statically patched files or memory dumps."
		hash = "0ff6abe0252d4f37a196a1231fae5f26"
	strings:
		$p1 = { 55 88 50 53 52 51 8D 64 24 94 DD 34 24 51 E8 ?? ?? ?? ?? 59 81 E9 14 00 00 00 8B 99 50 0F 00 00 83 FB 28 76 04 6A 31 }
		$p2 = { 59 81 E9 EE 00 00 00 6A 02 BB B4 05 00 00 01 CB C6 03 EB 43 C6 03 15 8B 44 24 78 83 C0 07 89 81 EC 07 00 00 E9 BF 02 00 00 }
		$p3 = { 50 53 52 51 E8 ?? ?? ?? ?? 59 81 E9 78 01 00 00 D9 99 C4 0F 00 00 8D 64 24 94 DD 34 24 FF B1 C4 0F 00 00 6A 02 EB 2D }
	condition:
		any of them
}

In this LABScon 25 presentation, Marc Rogers and Silas Cutler explore the complex, “shadow” supply chain of ultra-cheap Chinese smart home devices, specifically focusing on video doorbells and security cameras widely sold on mainstream online shopping platforms under various rotating brand names like Eken and Tuck.

Marc, who assisted the FCC Enforcement Bureau in its investigations, and Silas reveal how these devices often share identical hardware platforms powered by Allwinner semiconductors, a company heavily subsidized by the Chinese government.

Firmware analysis uncovered hardcoded root passwords and supposed security fixes that amounted to little more than commenting out vulnerable services from startup scripts rather than removing them. Despite appearing to use local cloud services, metadata and video content are frequently routed through servers in Hong Kong and China.

Rogers and Cutler trace a network of shell companies and fictional personas entirely absent from tax and voter records. These entities use non-responsive registered agents and PO boxes specifically set up to refuse legal service, effectively shielding the actual manufacturers from regulatory oversight and making enforcement nearly impossible.

The rapid iteration of hardware versions with no long-term support mirrors distribution patterns more commonly associated with malware campaigns.

While the investigation stops short of attributing direct malice, Rogers and Cutler argue that these devices collectively form a massive, vulnerable IoT surface that can be controlled through simple configuration pushes from overseas. Consumers are drawn in by low prices and subscription features, unaware that their data ultimately resides under foreign control.

About the Authors

Marc Rogers is Co-Founder and Chief Technology Officer for the AI observability startup nbhd.ai. Marc has served as VP of Cybersecurity Strategy for Okta, Head of Security for Cloudflare and Principal Security researcher for Lookout. In his role as technical advisor on USA’s “Mr. Robot” and the BBC’s “The Real Hustle”, he helped create on-screen hacks for both shows.

Silas Cutler is a Principal Security Researcher at Censys, with over a decade of experience tracking threat actors and developing methods for pursuit. Before Censys, he worked as Resident Hacker for Stairwell, Reverse Engineering Lead for Google Chronicle, and as a Senior Security Researcher on CrowdStrike’s Intelligence team.

LABScon 2026 | Call For Papers

Submission Deadline: June 19, 2026

LABScon is a unique venue for original research to be shared among peers. The benefit of an invite-only audience of researchers is that there’s no need for long preambles or introductions – speakers are encouraged to dive right into their technical findings.

• Original content only.
• Talks are 20 minutes long + 5 minutes for Q&A.
• Workshops are 90 minutes long.
• LABScon is primarily a threat intelligence and vulnerability research conference but we keep an open-mind.

About LABScon

This presentation was featured live at LABScon 2025, an immersive 3-day conference bringing together the world’s top cybersecurity minds, hosted by SentinelOne’s research arm, SentinelLABS.

Keep up with all the latest on LABScon here.

Executive Summary

• Large Language Models can perform static malware analysis, but individual tool runs produce unreliable results contaminated by decompiler artifacts, dead code, and hallucinated capabilities.
• We built a multi-agent architecture for reversing macOS malware that treats each reverse engineering tool (radare2, Ghidra, Binary Ninja, IDA Pro) as an independent, skeptical analyst in a serial pipeline, where each agent must verify or reject the claims of the previous one.
• We examine a concrete design decision: why we chose deterministic bridge scripts over the Model Context Protocol (MCP) for tool integration, and how this affects accuracy, latency, and token cost in production.
• We document the model routing strategy and some real-world challenges encountered during development.

Why Single-Tool LLM Analysis Fails

Anyone who has taken decompiler output, a string dump or raw disassembly from a binary, pasted it into an LLM, and asked “what does this do?” will recognise the failure mode. The model produces a confident, well-structured report that looks plausible until a human reviewer checks the virtual addresses and finds half the cited functions are wrong, several “capabilities” are actually dead code from the compiler’s standard library, and the claimed C2 endpoint has an extra character because the string extraction tool mangled a forward slash.

These failures are not hallucinations in the usual sense. The model is doing what it was asked to do, reasoning over the data it sees. The problem is that the data is noisy. Each reverse engineering tool brings its own parsing quirks. Radare2 string blobs can mangle delimiters; Ghidra’s decompiler might misclassify compiler stubs as application logic; IDA’s Hex‑Rays pseudocode can elide important register‑level details. If an LLM treats these outputs as ground truth, artifacts can make it into the final report that lead to erroneous “confirmed” capabilities.

Our experience has long taught us the value of using multiple tools to enrich our understanding of malware design and capabilities. Therefore, we set out not to try to build better prompts for our LLM agents, but rather to build a system where multiple tool artifacts are evaluated before they reach the report writing stage.

The Serial Consensus Pipeline

The system currently runs on OpenClaw, an open-source agent framework, and is built around a central Orchestrator agent that manages a team of specialized subagents, one for each reverse engineering tool plus a dedicated report-writer agent.

In our current deployment, all agents run on Anthropic’s Claude models: Opus 4.6 for the Orchestrator and report-writer, and Sonnet 4.6 for the subagents. The architecture is itself provider-agnostic, and OpenClaw’s design allows the operator to specify multiple fallback models in case the default models are unavailable or exhausted. However, compaction becomes a real issue once we start switching to smaller models like the Qwen2.5 32b that we configured as the ultimate ‘fail-safe’, and performance both in terms of response time and response quality can start to suffer with less capable models.

The pipeline operates in three phases. In the first phase, four tool-specific subagents run in sequence: r2, then Ghidra, then Binary Ninja, then IDA Pro. Each agent receives the accumulated findings from all previous agents, encoded in a structured document called the Shared Context. Each agent’s job is to run its specific tool against the binary, verify or reject the claims in the Shared Context, and add any new findings of its own.

Crucially, the Shared Context is an entirely in-memory construct. It is never written to disk during the analysis. When r2 finishes its analysis, its subagent outputs the Shared Context table as a conversational response back to the Orchestrator. The Orchestrator simply injects that exact text block into the prompt for the next subagent, controlling Ghidra. The LLM’s context window acts as the pipeline’s RAM, carrying the state of the analysis from one agent to the next until the final report is synthesized.

In the second phase, which we refer to internally as “the Gauntlet,” the same subagents run again in a different order, but this time they are explicitly tasked with peer-reviewing the assertions from the first round. Ghidra reviews IDA’s claims. Binary Ninja reviews Ghidra’s. IDA delivers the final verdict. Only findings that survive this adversarial review, or that present irrefutable evidence, proceed to the final stage.

In the third phase, the dedicated report-writer agent receives the finalized Shared Context and produces the output report, with every capability claim anchored to a specific virtual address and accompanied by a decompilation snippet.

The critical constraint is that the pipeline is serial, not parallel. Each agent sees what every previous agent has said, including what they rejected. This creates a cumulative evidence chain rather than independent votes.

The Active Rejection Mandate

The system prompts for the four tool-specific subagents include an explicit instruction to act as a “highly skeptical peer.” If Ghidra’s decompiler shows that a function flagged by r2 as a “decryption loop” is actually a compiler-generated string initialization stub, the Ghidra agent is not simply expected to note the discrepancy. It is instructed to formally reject the claim and document the reason.

This adversarial approach is enforced through the output schema. Every finding must include a Consensus field with a value of AGREE or DISAGREE, and rejected claims are tracked in a dedicated table in the Shared Context alongside the tool that rejected them and the rationale.

In practice, this mechanism caught a real artifact during our first pipeline run against an old SysJoker sample. Radare2’s string parsing rendered the C2 API endpoint as /api/req_res (with an underscore), while Ghidra’s decompiler correctly extracted the literal string from the data segment as /api/req/res (with a forward slash). In another test, the Gauntlet prevented the analysis from mistaking standard Go runtime strings for what was at first classified as a Tor .onion C2 address.

Without the rejection mechanism, these misinterpretations would have appeared in the final report. That kind of subtle corruption is exactly what makes automated reports untrustworthy, and precisely what the consensus pipeline is designed to prevent.

Similarly, the Gauntlet phase later caught a pure hallucination derived from a decompiler artifact in Binary Ninja’s Medium Level IL, which claimed the presence of a “download” instruction type. Because the agents reviewed each other’s work serially, this was actively rejected in the final report synthesis:

"Rejected claim R2: The command type 'download' does not exist in this binary. 
The strings 'exe' and 'cmd' are the only type discriminators. 
The 'download' string was a Binja MLIL decompiler artifact."

The adversarial design also helps solve the problem of different disassembler and decompiler output, with tools able to be evaluated against each other in real-time. In one of our tests, only Ghidra initially found the XOR-obfuscated strings in a WizardUpdate sample, but the others were able to confirm the finding once told to specifically weigh in on whether the Ghidra subagent was right or just hallucinating.

The Token Economics of Consensus

Running up to seven subagents per binary sounds computationally expensive, but the serial architecture creates an asymmetric token load that prompt caching handles exceptionally well.

The image above shows the Orchestrator managing Round 2 (the Gauntlet). Note the drop in token consumption as the analysis shifts from raw extraction to peer review. During Round 1, the agents consume significant context. A raw IDA Pro disassembly dump can push a subagent’s token count past 100,000.

However, because we use deterministic bridge scripts that dump each tool’s entire output to disk rather than interactive MCP endpoints that require sequential back-and-forth prompting, this represents a single massive context load. The evolving Shared Context state is injected dynamically on top of this static tool output, so the underlying tool data remains mathematically constant. According to Anthropic, prompt caching delivers “up to 90%” lower input costs and 85% lower latency for long prompts, making repeated use of large static tool outputs less expensive in practice.

More importantly, the token burden drops drastically during Round 2. When the Orchestrator spawns binja-r2-gauntlet for peer review, the subagent is no longer parsing the raw disassembly. It is only evaluating the distilled Shared Context document against specific contested claims, dropping its token consumption by more than half (approx. 44,000 tokens). The data has been refined, making the adversarial consensus phase both faster and cheaper.

Bridge Scripts Over MCP

One of the first architectural questions was whether to use the Model Context Protocol (MCP) as the interface between the LLM agents and the reverse engineering tools. IDA Pro, for example, has an existing MCP server that allows an LLM to interactively query the disassembly database: requesting the decompilation of a specific function, querying cross-references, renaming variables, and so on.

MCP is designed for interactive, human-in-the-loop workflows where an analyst works alongside an AI copilot. For fully automated batch analysis, it introduces two significant concerns.

The first is latency. An MCP-based agent must make sequential API calls to explore the binary, then request cross-references for a given function of interest, then another call to,  say, query the strings in .rodata. Each call requires a round-trip to the LLM to decide what to ask next. A typical function-level analysis might require 15 to 50 MCP tool calls. In a pipeline with seven subagent invocations across two rounds, this would compound into considerable wall-clock time and token cost.

Even if those weren’t an issue, the second problem is non-determinism. Because the LLM decides what to query, it can and will miss things. If the agent does not think to ask about cross-references to a specific crypto constant, it will not discover the decryption routine. A deterministic bridge script, by contrast, is programmed to extract everything: all strings, all imports, all cross-references, all function signatures, in a single sweep, regardless of whether the LLM would have thought to ask for them.

In our design, we built thin bridge scripts, one per tool, that invoke each tool’s headless analysis mode and dump comprehensive output to a text file. The bridge for IDA Pro, for example, is a 40-line shell script that calls idat64 in batch mode with a universal IDAPython analysis script. The bridge for Binary Ninja is a Python wrapper that invokes the Binary Ninja API in headless mode.

# The IDA bridge: core execution and error handling
"$IDAT_PATH" -A -B -S"$UNIVERSAL_SCRIPT" -L"$OUTPUT_DIR/ida_analysis.log" "$BINARY"
EXIT_CODE=$?
if [[ $EXIT_CODE -ne 0 ]]; then
  echo "ERROR: IDA Pro analysis failed with exit code $EXIT_CODE" >&2
  exit $EXIT_CODE
fi

The trade-off here is that while we lose the interactive exploration capability that MCP provides, we gain deterministic, comprehensive extraction with predictable latency. For an automated pipeline leveraging probabilistic inference machines, our view is the trade-off strongly favors the bridge approach.

Tiered Reasoning Across the Pipeline

Not all tasks in the pipeline require the same level of reasoning. The Orchestrator must synthesize conflicting findings, decide what to reject, and construct structured handoff prompts. A subagent, by contrast, has a narrower job: parse tool output, fill in a schema, and flag disagreements.

We configured the system to use a stronger model for the Orchestrator and report-writer (the two highest-reasoning roles) and a faster, cheaper model for the four tool-specific subagents, where the task is essentially structured extraction from well-formatted decompiler output. OpenClaw supports this through its agents.defaults.subagents.mode configuration, which sets a default model for all spawned subagents independently of the main agent’s model.

The cost implication is that seven of the nine LLM invocations in a full pipeline run use the less expensive model, while the two highest-value calls (orchestration and report synthesis) use the stronger one. In practice, this produces a roughly 30% to 50% cost increase over a single-model configuration using the less expensive model, but it is a cost that buys us a disproportionate improvement in report quality. The stronger model is better at detecting when a subagent finding contradicts an earlier one, and better at maintaining the strict output formatting required by the report template.

However, there is a practical constraint to this approach. The stronger model has tighter rate limits, and during our initial testing, we found that API congestion caused the Orchestrator to fall back to the secondary model mid-run. To avoid saturating the provider’s rate ceiling, we reduced the main agent concurrency cap from four to two. The next section describes how this played out during the first full pipeline run.

Lessons From the Early Runs

To test our design, we began with a known Mach-O sample of the SysJoker malware. Using a known sample allowed us to evaluate the LLMs output against that of several human analysts and public reporting. The initial full pipeline run surfaced several issues that were not visible during isolated testing of individual components.

The most disruptive early issue was duplicate session handling. Due to display issues in OpenClaw’s TUI, we chose to drive the analysis through its open source Web UI. A browser automation glitch caused three identical analysis requests to be submitted simultaneously, each of which spawned its own complete pipeline. The resulting load triggered API rate limiting, causing the Orchestrator to fall back to the secondary model, and creating multiple competing report-writer sessions trying to produce the same output. The architectural fix was to cap the main agent’s concurrency limit, reducing it from four to two, but the debugging cost both time and a non-trivial number of API tokens.

However, this rate-limit congestion also proved the resilience of the Orchestrator model. During one test run, a subagent worker thread was silently killed by an upstream API timeout midway through the pipeline (specifically, the final report-writer was lost during the model handoff). Because the Orchestrator maintains the entire accumulated state in its conversational history rather than delegating it to the subagents, the analysis did not crash.

When we prompted OpenClaw that the report had not arrived, the Orchestrator simply observed that the subagent had stopped responding, preserved the Shared Context from the previous round, and explicitly commanded a respawn of the dead subagent to continue the pipeline. By decoupling state management (the Orchestrator) from computation (the subagents), the system is capable of resuming the task and avoids wasting tokens or entire runs starting from scratch.

A subtler issue was output schema inconsistency across the four specialist skills. We initially had minor differences between them: radare2’s output schema lacked a Consensus field since it runs first and has nothing to compare against, and some skills included a two-line safety block while others had only one line. These small differences created parsing ambiguity for the Orchestrator when it attempted to align findings across tools. The fix was to normalize all four schemas to be structurally identical, with r2 using Consensus: N/A - First Pass as a placeholder value.

The Orchestrator’s handoff format also required explicit definition. Initially, without a specified Shared Context schema, the LLM would invent its own handoff format for each subagent, making inter-agent communication fragile and difficult to parse programmatically. We defined a strict markdown table format with markers (SHARED_CONTEXT_START / SHARED_CONTEXT_END) and three categorized tables: Verified Capabilities, Flagged for Review, and Rejected Claims. This made the inter-agent communication deterministic enough for the Orchestrator to reliably merge findings across rounds.

Finally, bridge scripts needed explicit failure handling. When the underlying tool failed (for instance, if IDA could not import the binary), the original scripts printed “Analysis complete” regardless of the exit code. The subagent would then attempt to parse an empty output file and produce nonsensical findings. Adding exit code propagation, where a non-zero tool exit terminates the bridge with a clear error message, gives the Orchestrator a reliable signal to handle the failure rather than proceeding with garbage input.

Conclusion

The primary challenge with LLM-driven malware analysis is not so much a given model’s reasoning capability but the quality of the data the model reasons over. Decompiler artifacts, string parsing quirks, and dead code all create noise that an LLM will faithfully amplify into a report unless the system is specifically designed to catch and reject those artifacts before they reach the synthesis stage.

The multi-agent consensus pipeline described here is one approach to that problem. By treating each reverse engineering tool as an independent analyst with an explicit mandate to challenge the claims of other tools, the system produces reports where every capability is backed by cross-validated evidence anchored to specific virtual addresses.

The architecture is intentionally simple: bridge scripts extract data, subagents evaluate it, the Orchestrator synthesizes consensus. There is no vector database, no fine-tuning, and no custom model. The reliability comes from the pipeline structure, the serial handoff, the rejection mandate, and the structured Shared Context, not from the model itself.

Sample Hashes

60c8128c48aac890a6d01448d1829a6edcdce0d2 WizardUpdate
678aa572faa73f6873d24f24e423d315e7eb2c2d Go Infostealer
ad7d2eb98ea4ddc7700db786aadb796b286da04 FinderRAT
f5149543014e5b1bd7030711fd5c7d2a4bef0c2f SysJoker

In this LABScon 25 talk, Andrew MacPherson dives deep into the high-stakes world of crypto crime, which has amassed approximately $9 billion in illicit funds. Andrew demystifies the technical landscape and exposes the sophisticated attack vectors plaguing the decentralized finance (DeFi) space.

The talk begins with an explanation of the core concepts necessary to understand crypto-related security threats, including definitions of blockchains, wallets, and smart contracts. Andrew explains that a key point in the architectural difference of many crypto applications is that they typically rely solely on frontends, with all interactions happening in the browser via the wallet extension.

The talk then moves on to focus on attack patterns. Crypto thieves target every weak point, from applications and code to the developers and executives themselves. The speaker details the largest crypto heist to date, the $1.5 billion loss from Bybit. This attack involved infecting a developer’s machine, gaining access to production JavaScript code, and modifying it to authorize a full wallet drain during a multi-signature transaction. The talk also covers supply chain risks like typo-squatting, exploitation of personal servers like Plex to compromise GitHub accounts, and the rise of “drainers as a service” that simplify crypto theft.

Andrew also covers the challenges attackers face in laundering stolen funds, and how they leverage techniques such as cross-chain swaps, using mixers like Tornado Cash, and non-KYC platforms for conversion to cash. Despite the fact that all blockchain logs are public and permanent, the presentation also discusses the challenges threat intel analysts face in tracking these rapidly moving funds.

Andrew’s presentation is essential viewing for anyone interested in cryptocurrency and cybersecurity, especially those looking to understand the technical realities of financial crime in the decentralized era.

About the Author

Starting at Paterva, Andrew Macpherson spent more than 10 years creating Maltego before moving to the US for security roles at BitMEX (IR), Robinhood (IR/D&R), Uniswap (Head of Security), and now Privy (Principal Security Engineer). He’s spoken at Black Hat, DEF CON, DSS, EthCC and countless others, teaching courses and drinking malibu on the way.

About LABScon

This presentation was featured live at LABScon 2025, an immersive 3-day conference bringing together the world’s top cybersecurity minds, hosted by SentinelOne’s research arm, SentinelLABS.

Keep up with all the latest on LABScon here.

Overview

In this blog post, we explore the application of large language models (LLMs) for extracting and contextualizing information from cyber threat intelligence (CTI) reports, turning narrative into structured data for downstream use.

As part of our broader continuous innovation in automating defense workflows with AI, this work focuses on the use of LLMs for information extraction in the CTI domain, outlining relevant insights, key challenges, and trade‑offs involved, supported by empirical evaluations. It is intended to support CTI teams and cyber defense organizations considering the development or adoption of AI‑enabled CTI information extraction capabilities.

CTI reports contain rich information about adversary behavior, infrastructure, and intent. For defenders, they provide timely insights into ongoing campaigns and evolving techniques, helping teams keep pace with the current threat landscape, prioritize detections, and accelerate their response to novel threats. However, because this information is conveyed in narrative form, the manual extraction of relevant elements such as indicators of compromise (IOCs) and contextual details is slow, inconsistent, and difficult to scale. LLMs have the potential to automate this task by interpreting narratives, extracting explicit data, and inferring implicit relationships, transforming text into structured, machine‑readable data that supports defense workflows at all levels of automation.

AI for Extracting and Contextualizing CTI Information

Non‑LLM‑based methods, such as pattern‑matching approaches, can automatically and accurately extract explicit elements that follow well‑defined formats, for example atomic IOCs like IP addresses or file hashes. However, LLM‑driven extraction can be applied in more complex scenarios that require semantic understanding or adaptable inclusion criteria beyond simple pattern recognition.

Certain use cases may demand selective IOC extraction, for example by focusing on attacker‑registered domains while filtering out benign ones. Such distinctions often depend on contextual cues within the report, such as whether a domain is described as adversary‑registered infrastructure or simply mentioned in passing during the description of normal network behavior.

In addition, capturing the broader context that extends the operational value of atomic IOCs, such as infrastructure ownership, compromise state, their association with specific threat actors, victimology, and characteristic TTPs, or the role within an intrusion chain, remains a demanding challenge for non‑LLM approaches. Unlike atomic IOCs, these contextual details are often implicit rather than explicitly stated and therefore must be inferred.

This context is increasingly important as the standalone value of atomic IOCs continues to diminish in an era of rapid shifts in adversary techniques, tools, and infrastructure. It is important for guiding accurate and effective detection and response decisions involving the associated IOCs, as well as for other purposes, such as uncovering related malicious activities through context‑aware threat hunting and developing detections that retain value beyond individual observables.

Beyond improving detection, response, and threat hunting, context‑enriched intelligence extracted by LLMs can integrate with organizational defense systems such as threat‑intelligence platforms (TIPs) to support collaboration, correlation, prioritization, and organization‑wide sharing of intelligence. This integration transforms CTI narrative into structured, linked knowledge that can be leveraged across organizational defense workflows.

Scope and Structure

The extraction of information from CTI narratives using LLMs has been explored in previous research. This blog complements existing work by taking a practical perspective, focusing on selective IOC extraction, the structured representation of contextual information, and the automated reconstruction of adversary activity into playbook-level sequences with inferred chronology. It presents a preliminary study intended to demonstrate feasibility and highlight key design, evaluation, and operational considerations in designing LLM‑driven CTI information extraction systems.

The aim is to share practical insights drawn from our own experience rather than to rank individual models or propose a complete solution. To illustrate these points, we use a basic, preliminary setup as a running example, and evaluate general‑purpose LLMs to measure out‑of‑the‑box performance in extracting information.

The following sections outline our approach along with the evaluation methodology and results. We begin by describing the overall workflow and data structures used for information extraction, followed by the method used to instruct and guide the LLM during extraction. We then present the evaluation setup and discuss results across multiple dimensions, including extraction performance, processing efficiency, and output quality.

Information Extraction | Workflow Overview

Our preliminary workflow for extracting information from CTI reports consists of three phases.

Phase 1 | Report Ingestion and Sanitization

In Phase 1, the Sanitizer component ingests reports in HTML format and removes non-content elements such as navigation (nav), headers, footers, sidebars (aside), scripts (script), and styles (style). It then converts the remaining content to clean, plain text, preserving the text of headings, paragraphs, lists, and tables, while discarding the HTML markup. This reduces noise, standardizes inputs across sources, and lowers the risk of errors in downstream extraction.

Phase 2 | LLM‑Based Extraction

In Phase 2, the sanitized report content is passed to LLM-based extractors, the Infrastructure, Executables, and Playbook Extractor. They use an LLM to reason over the input text in order to extract information and produce structured output. Each extractor’s LLM is guided by:

• an extractor-specific output data model that defines entities and associated attributes, data types, and inter-entity relationships;
• LLM instructions that define the extraction policy, including value assignment criteria for fields defined in the data model.

The output of each extractor is a JSON record generated according to its data model, effectively turning narrative text into structured, machine-readable data.

Each extractor’s data model is an in‑house specification tailored to its analytical scope. For example, the Infrastructure and Executables Extractor use data models that define atomic IOC types, specifically Infrastructure (domain or IP) for the Infrastructure Extractor and Hash (MD5, SHA-1, or SHA-256 hash) for the Executables Extractor, as well as per-type IOC contextual attributes. Together, these models define 12 IOC contextual attributes, typed either as enumerations (categorical labels) or as open‑text strings. The categorical attributes capture aspects of the operational context of network and executable artifacts, such as their functional roles within threat actor operations (usage), their behavioral properties (injection), and the points in time or intrusion stages at which they are employed (attack_stage). The open-text attributes capture explicitly reported details extracted from the input text, such as file paths (filepath), command‑line arguments (cmdline), or names of injected processes (injected_processes).

Across the extractors’ data models, the categorical attributes are implemented as tri‑ or four‑state variables that standardize annotation. For example, they allow the LLM to distinguish between positive and negative evidence in binary contexts (true and false), assign a composite value when evidence supports more than one allowed category (both), and denote cases where evidence is absent or insufficient for classification (None).

To instantiate data model entities, the LLMs of the Infrastructure and Executables Extractor first selectively extract atomic IOCs and then assign values to the associated contextual attributes for each extracted IOC by performing:

• classification for categorical attributes, assigning a value from a predefined set of allowed labels;
• text extraction for open‑text attributes, assigning unconstrained string values taken directly from the input text.

The selective extraction of network‑related atomic IOCs (domains and IPs) focuses on indicators that have played an active role in the adversary activity described in the input report, encompassing both attacker‑owned assets and compromised external systems deliberately used for malicious purposes. The intent is to capture infrastructure that has directly supported the activity while excluding references to resources that fall outside the actor’s control, such as legitimate infrastructure mentioned during the description of normal network behavior. The selective extraction of hashes focuses on those associated with malicious or attacker‑used software components that have been deployed within victim environments, including both custom malware and publicly available tools used for offensive purposes. It excludes hashes corresponding to legitimate third‑party binaries that appear in attack chains, for example, as part of DLL hijacking or other forms of abuse.

The LLM of the Playbook Extractor instantiates data model entities as follows:

• Extracts distinct threat actor actions, represented by Step entities, and groups them into one or more Playbook entities. A Playbook represents a sequence of Step entities within a single adversary operation or campaign. Separate Playbook entities are created when the report shows explicit or clearly implied operational separations, for example, distinct campaign names, non‑overlapping timeframes, different targets or regions, or unrelated objectives.
• Infers the chronological order of the Step entities within each Playbook and creates directed relationships between them to reflect the inferred sequence.
• Maps each Step to the appropriate MITRE ATT&CK tactic, technique, and, where applicable, sub-technique recorded in the associated TTP entity. The procedure attribute of TTP records the distinct threat actor action identified earlier.
• Extracts contextual information about the threat actor attributed to each Playbook (for example, actor name, aliases, country of origin, and motivation) and links a ThreatActor entity representing the actor to the corresponding Playbook entities.

The Playbook Extractor constructs a directed acyclic graph over ThreatActor, Playbook, Step, Infrastructure, and Hash entities by adding links according to sequencing and linking rules, with the goal of producing self-contained flows with consistent chronology, valid MITRE ATT&CK mappings, and coherent relationships.

Phase 3 | Knowledge Graph Assembly

In Phase 3, the LLM of the Playbook Extractor maps each atomic IOC and its contextual attributes (an Infrastructure or Hash entity instantiated by the Infrastructure or Executables Extractor) to threat actor actions that use or produce it; that is, establishes INVOLVES relationships from Step entities. This combines all extracted information into a unified knowledge graph, which serves as the final consolidated output for downstream applications.

Task Granularity and Input Modality

Each LLM‑based extractor performs multiple distinct subtasks. For example, the Infrastructure and Executables Extractor selectively extract atomic IOCs and interpret the input text to assign values to IOC contextual attributes, a task that requires context‑sensitive, evidence‑based inference. The Playbook Extractor handles an even more complex set of operations, including identifying distinct threat actor actions, inferring their chronological order, mapping them to MITRE ATT&CK tactics, techniques, and sub-techniques, and linking any attributed threat actors and IOCs to the relevant actions.

Depending on task complexity and the diversity of reasoning steps required, LLMs may find multi‑objective workloads challenging, as attention and inference capacity are distributed across diverse goals. Dividing broad tasks into smaller components may sharpen focus, preserve contextual consistency, and improve extraction and classification quality. However, segmentation adds orchestration complexity, increases the risk of error propagation across subtasks, and may introduce latency. Choosing the right granularity requires balancing the gains against these costs.

Beyond task segmentation, input modality and coverage are also important for extraction outcomes. In our preliminary implementation, the LLM‑based extractors operate only on textual content, leaving a gap in coverage for reports that embed relevant text in images, such as command lines, tool outputs, or malware code snippets, which is common in CTI reporting. Using optical character recognition (OCR) to extract text from images in reports can make this information accessible to LLM‑based extraction workflows and enrich the input with observables and contextual details that would otherwise be lost. However, OCR introduces trade‑offs that should be accounted for, as transcription errors, noise from low‑quality or stylized visuals, and inconsistencies in extracted text formatting can complicate downstream processing.

Data Model Selection and Design

Industry‑standard data models, like STIX, provide broad interoperability and predefined representations for common CTI concepts such as atomic IOCs, malware, threat actors, campaigns, and techniques, and the relationships among them. Adopting or extending an existing standard as the output data model in CTI extraction workflows is particularly appropriate for organizations that require compatibility with external feeds or cross‑organizational sharing platforms. This approach is also suitable when the cost of designing and maintaining a custom in‑house model cannot be justified.

In contrast, a custom data model can be more effective when extraction and analysis serve internal, organization‑specific needs rather than cross‑organizational sharing. Such a model provides full control over scope and structure, allowing its size and complexity to match current requirements. This flexibility supports precise extraction of the information relevant to specific use cases without being constrained by predefined relationships, hierarchical elements, or granularity imposed by external data models, which may be unnecessarily complex for certain applications. For example, data model organization, such as the depth of elements within the structural hierarchy, carries semantic weight that shapes LLM inference, making deliberate design choices important.

Our case exemplifies this scenario. Each extractor uses a custom output data model optimized for internal analytics and tailored to integrate with proprietary formats of endpoint telemetry and other log data. This alignment supports specific internal applications such as proactive threat hunting and telemetry enrichment. For example, the top-level IOC contextual attribute is_compromised distinguishes infrastructure intentionally set up by the attacker from legitimate but compromised assets. This distinction can make the difference between targeted action and unintended disruption. If a domain observed in our telemetry is described in a processed CTI report as attacker-controlled infrastructure, traffic can be blocked and related domains identified by pivoting on attributes such as certificate fingerprints or registration data. In contrast, if the domain is described as belonging to a compromised legitimate website, blocking policies can be applied in a way that minimizes interruption to legitimate services, favoring precise and reversible measures such as URL‑specific filtering and follow‑up verification before relaxing controls.

An important aspect of designing an output data model for CTI extraction workflows is the linguistic formulation of elements such as field names and categorical labels. Just as hierarchy depth carries interpretive significance, wording can influence how LLMs allocate attention during classification and text extraction. Because LLMs rely on natural‑language context to guide reasoning, phrasing choices for field names and category labels implicitly frame how evidence is interpreted and may cue different decision boundaries, potentially biasing a model toward particular outcomes. For example, we have observed measurable differences in classification accuracy when LLMs assign values to categorical IOC contextual attributes under different phrasings of field names and labels. Using terminology that guides LLM reasoning toward the intended interpretation and decision boundaries helps ensure that the LLM’s attention and inference are aligned with the extraction objectives.

Information Extraction | LLM Instructions

The LLM‑based extractors operate using structured prompts with domain‑specific instructions aligned with their respective data models. For example, guided by these prompts, the Infrastructure and Executables Extractor analyze the entire input text, selectively extract atomic IOCs that meet the defined criteria, and assign values to contextual IOC attributes based on the available evidence.

Each extractor operates within defined reasoning boundaries. Across all extractors, the prompts combine extractor‑specific task scopes with a unified reasoning policy, which defines how the LLM interprets evidence. They constrain the model’s reasoning to explicit and strongly implied evidence, with the degree of inference bounded by an evidence‑grading scale. Each extraction decision is graded by evidence strength: High for explicit statements, Medium for strongly implied information, and Low for weak or speculative cues, which the LLM is instructed to ignore.

Building on this evidence‑grading approach, a unified decision‑making framework ensures consistent logic in how values are assigned across all categorical fields. The framework also defines how multiple candidate values are resolved for the same field and specifies conflict‑resolution procedures for reconciling competing or contradictory evidence.

The following prompt excerpt, shown in Markdown format, illustrates some of the evidence grading and decision‑making principles.

### Evidence and inference
- Use only information that is explicitly stated or strongly implied in the report. Weak, associative, or speculative cues must not be used for classification.
- Evidence confidence levels:
- High: explicit statements or direct behavioural descriptions.
- Medium: strongly implied and supported by multiple consistent observations.
- Low: weak or speculative — ignore low‑confidence signals when assigning values.
- Absence of evidence for one label is not evidence for another.

### Decision‑making framework
(Applies to all fields including tri‑state and multi‑class labels; e.g., `{'true','false','None'}`)
 	1. Identify candidate labels (set C) = labels in the field’s allowed set (excluding 'None') that have explicit or strongly implied evidence according to the field definition.
 	2. If C is empty: Set the field to 'None' (the report lacks qualifying evidence for any label).
 	3. If |C|=1: Set the field to that label.
 	4. If |C|>1:
		- If the field defines a valid composite/union label (e.g.,'both') and evidence supports all involved roles on the same artifact → assign the union label.
		- Otherwise apply the field’s specific precedence/conflict rule.

Prompt Optimization and Conceptual Boundaries

The effectiveness of the instructions in guiding the LLM‑based extractors to accurately extract information depends not only on prompt design but also on the models that interpret them. Because model versions and families differ in how they represent and interpret language, infer meaning, and translate instructions into reasoning steps, the same prompt can produce model‑specific differences in interpretive and response behavior.

Prompts can be optimized for the reasoning patterns and instruction‑following behavior of a specific model, which in the context of this work can improve information extraction quality. However, model‑specific prompt optimization increases maintenance overhead when models are frequently updated or replaced. For example, in managed environments where older model versions may be deprecated over time, each update requires not only prompt adjustments but also a reevaluation of the prompt’s effectiveness before deployment.

In addition to model-specific prompt optimization, defining the specific semantic scope of categorical data model fields that encode analytical concepts is an important yet challenging aspect of designing effective LLM instructions for information extraction. These scope definitions determine how these fields translate complex real‑world operational behavior and relationships into discrete categories that the model can apply consistently. They function as a layer of conceptual modeling that mediates between the descriptive language of reports and the structured reasoning required for extraction, encoding within the instructions the definitional decisions that establish each field’s scope. Well‑defined scopes enable consistent model interpretation and coherence across downstream analytical processes, whereas vague or internally inconsistent boundaries lead to misclassification and undermine overall reliability.

In our case, clear scope definition is particularly important for the categorical IOC contextual attributes, which require deliberate, analytically grounded boundaries. For example, defining the scope of usage, which distinguishes infrastructure used for command‑and‑control from that used only to host malicious content or store exfiltrated data, requires giving the model clarity on multiple concepts, including what constitutes command‑and‑control, malicious content, and passive hosting.

Evaluation Setup

In the following sections, we present the results of an evaluation study of several off‑the‑shelf, general‑purpose language models from OpenAI and Anthropic — GPT‑4.1, GPT‑5, GPT‑5.2, Claude Sonnet 4.5, and Claude Opus 4.5 — used within the Infrastructure, Executables, and Playbook Extractor without any additional fine‑tuning or task‑specific adaptation. We quantified performance from multiple complementary perspectives using the same set of extractor prompts across all models.

Where applicable, the reasoning mode for the GPT models was set to High, the Claude models were configured with a thinking budget of 16000 tokens to allow for extended reasoning, and the LLM temperature parameter was set to 0 to minimize randomness and reduce non‑deterministic behavior.

The reported results are preliminary and based on a limited ground truth dataset comprising 343 atomic IOCs and 1859 labeled IOC contextual attribute instances. The purpose of this evaluation is not to provide conclusive performance comparisons but to demonstrate the feasibility of using LLMs for information extraction from threat intelligence narratives.

To account for the inherent non-determinism of LLMs and to provide statistically reliable results, all reported metric values were obtained through repeated executions of each evaluation experiment until the point estimates reached a 95% confidence level with a relative precision of less than 5%.

Manual Ground Truth Creation

There is no readily available common ground truth dataset that enables evaluation of LLM performance in extracting information from CTI reports. A dataset suitable for this purpose must be aligned with the LLMs’ expected outputs, reflecting the same data model, field definitions, and scope boundaries used during extraction to enable direct comparison between LLM predictions and reference data. Differences in analytical focus and output data model design across potential CTI information extraction approaches effectively preclude the possibility of a common ground truth dataset. Even though standardized data models such as STIX could, in principle, support a shared ground truth dataset, implementations can add custom extensions to accommodate organization‑specific analytical needs, reintroducing differences in data model design and scope.

Creating a ground truth dataset aligned with the output data model and analytical scope of a given CTI information extraction approach is a time‑consuming process that relies on manual annotation guided by expert judgment to ensure accurate interpretation of CTI reports and consistent application of field‑scope definitions and label criteria.

Ground Truth for Ambiguous Evidence

When extracting and classifying information from CTI reports into discrete values, both human analysts and LLMs face the inherent ambiguity of natural language reporting. CTI reports vary widely in precision and contextual completeness, meaning that informative cues supporting a given interpretation may be partial or implied rather than explicit. For example, an IP address listed in a generic IOC table might appear without any narrative cues describing its operational use. In such cases, the value of the usage attribute in our data model becomes uncertain: one annotator may assign C2 if the report primarily discusses adversary C2 infrastructure, whereas another, applying stricter evidentiary standards, may assign None, indicating insufficient evidence to support any other specific label. Neither interpretation is necessarily incorrect; they reflect differing thresholds for inference, with one adopting a looser contextual assumption and the other following a stricter evidence‑based criterion.

The adequacy of contextual detail in CTI reports depends on the type of information being extracted, how much inference is allowed to bridge contextual gaps, and other factors, including the report’s intended audience and analytical scope. For example, when extracting information about technical artifacts, strategic reports aimed at broad audiences may lack the specificity needed for reliable extraction, whereas reports written for technical analysts are more likely to include the context required to support such extraction. Beyond the quantity of contextual detail, ambiguity can also result from linguistic and structural sources of uncertainty in CTI reporting, such as inconsistent terminology, implicit assumptions, and condensed summaries.

While a highly conservative strategy can be applied, allowing minimal interpretive flexibility and constraining extraction to cases only where very explicit supporting evidence is present, such rigidity may be impractical. If most of the information to be extracted depends on ambiguous evidence, the overall volume of extracted information would become severely limited.

Considering the inherent limitations and ambiguities of CTI reporting, even experienced human analysts, who are afforded a degree of interpretive flexibility, may assess the available evidence differently, with some adopting broader interpretations while others adhere to stricter criteria. LLMs granted comparable interpretive flexibility show similar indecisiveness when confronted with ambiguous or incomplete information, producing outputs that mirror the uncertainty observed in human reasoning.

To ensure accurate evaluation of LLMs that extract information from CTI narratives with some interpretive flexibility, the ground truth datasets should account for these ambiguities. For genuinely underspecified cases, it may be more realistic to define multiple acceptable values rather than a single correct label. However, developing such flexible ground truth increases the labeling effort: ideally, for each ground truth element where the correct value may be ambiguous, multiple human annotators independently assess the evidence and then reach consensus on whether the ambiguity is genuine and which alternative values are plausibly supported. This procedure captures genuine uncertainty without compromising methodological rigor.

The ground truth dataset we use in our evaluation study applies this flexible labeling approach, allowing multiple values to be considered correct for truly ambiguous cases.

As a reminder, the Infrastructure and Executables Extractor apply controls when assigning values to IOC contextual attributes that constrain how evidence is evaluated and how conflicting or insufficient cues are resolved. These controls limit classification to explicit or strongly implied contextual evidence and apply field‑specific rules that default to None when no support for another value is found. Even with these controls and deterministic inference settings applied where applicable (temperature = 0), models can still produce different label assignments across repeated runs when the input evidence is ambiguous. Such variation arises not from stochastic sampling but from minor non‑deterministic aspects of inference, such as floating‑point rounding or context‑evaluation differences, which slightly alter internal probability weighting. When a case lies near a conceptual decision boundary, between sufficient and insufficient evidence, or between competing interpretations supported by similar cues, these micro‑variations can shift the balance enough for the model to favour a different plausible label. Across models, these effects combine with differences in calibration of what constitutes strong, sufficient, or insufficient evidence, producing similar alternation among valid values under the same policy.

To illustrate this tendency of LLMs to alternate between valid values in ambiguous cases, we measured internal decision consistency for each evaluated LLM on two IOC contextual attributes for which different values were frequently accepted as valid by expert annotators. Specifically, we calculate two metrics for is_compromised and attack_stage, with the reported values conditioned on each model’s extracted IOCs:

• IOCs with Pₒ < 1: The proportion of extracted IOCs for which, across repeated runs under identical conditions, the LLM assigned two or more different values for the same attribute, alternating among the values that the expert annotators defined as valid (yielding observed agreement Pₒ < 1 across runs). This metric indicates how often the model switches among valid values under identical conditions.
• Average mode‑based observed disagreement D̄ₒ: For the subset of IOCs with Pₒ < 1, the average proportion of the LLM-assigned attribute values across the repeated runs that differ from the dominant (mode) value. This metric quantifies the degree of variability in the model’s assigned values across those runs.

Together, these metrics describe each model’s sensitivity to ambiguous or borderline inference conditions. Higher percentages of IOCs with Pₒ < 1 indicate greater fluctuation in how the LLM interprets ambiguous evidence, while higher D̄ₒ values show that, in cases where the model switches between valid attribute values, its decisions are more evenly distributed among the alternatives rather than converging on a single dominant interpretation.

These observations highlight why allowing multiple valid values in the ground truth data is important when evaluating LLMs that extract information from CTI narratives with some interpretive flexibility. Recognizing and encoding the ambiguity inherent in CTI reports ensures that evaluation reflects the realistic bounds of human interpretation rather than enforcing artificial certainty. The same principle should extend to downstream applications, where processes or systems consuming LLM outputs should be able to accommodate alternative but defensible value assignments.

Evaluation | Selective IOC Extraction

This section presents the performance of the evaluated LLMs in selective IOC extraction, measured using F1‑scores that capture the balance between precision (correctness of extracted atomic IOCs) and recall (extraction completeness under the predefined selection criteria). The reported values represent the average of the F1‑scores achieved by each LLM when used in both the Infrastructure and Executables Extractor, providing a single performance measure per model.

Report Structure and Formatting Effects

Variation in the formatting and structural presentation of IOCs, as well as in the availability of labeling and contextual cues such as column headers or textual indicators linking IOCs to relevant entities such as threat actors, malware, or campaigns, was a key factor contributing to differences in F1‑scores. CTI documents differ widely in how they present information, combining narrative text, tables, lists, and other structured elements with varying levels of detail and contextual labeling.

For example, some reports present IOCs in visually dense formats, such as tables listing multiple hash representations in a single row. These cases require the model to interpret logical relationships within structured data, for example how corresponding values relate across columns. This involves a degree of relational reasoning that some models apply inconsistently, particularly when labeling or contextual cues are absent or ambiguous, leading to missed indicators and reduced recall.

This observation highlights how the structure and formatting of CTI reports directly influence LLM extraction performance. Simplicity in IOC presentation, together with explicit labeling and unambiguous contextual cues, helps LLMs extract IOCs more accurately and consistently while maintaining interpretability for human analysts.

Evaluation | Report Processing Time

The charts below compare the average report processing times and the corresponding speed‑ups achieved by the Infrastructure and Executables Extractor configured with each evaluated LLM, alongside the baseline time required by human analysts. Report processing time refers to the end‑to‑end duration required to process a CTI report, including ingestion, reasoning, selective IOC extraction, IOC attribute value assignment, and output generation.

The metric represents the average time per report in minutes, rounded to the nearest half minute, and the speed‑up values express the same results relative to human processing time. The reported values represent the combined per‑report average processing time from both extractors, with the human baseline reflecting the equivalent manual processing of both extraction tasks, and are conditioned on each LLM’s extracted IOCs.

In all cases, the use of LLMs substantially reduced report processing time compared with human analysts, whose average was 41 minutes per report. On average, the extractors required about 3.3 minutes per report, corresponding to an aggregate speed‑up of more than 18 times. Even the slowest LLM-based setup processed reports approximately 6 times faster than the human baseline, while the fastest reduced average processing time by more than 97% relative to the human baseline. These results highlight the considerable time‑efficiency gains achieved by using LLMs for CTI information extraction compared with traditional human workflows, though with accompanying trade‑offs in extraction completeness and correctness.

Evaluation | Accuracy and Precision

The chart below reports accuracy and precision in assigning values to IOC contextual attributes for each evaluated LLM when operating within the Infrastructure and Executables Extractor:

• Standard accuracy: the mean of accuracies computed per IOC contextual attribute.
• Balanced accuracy: the mean of balanced accuracies computed per IOC contextual attribute; for each attribute, balanced accuracy is the average recall across value classes (for categorical attributes, the predefined labels; for open-text attributes, None vs any assigned value), which accounts for differences in value‑class distributions in the ground truth.
• Mean macro precision: the mean of macro precision values computed per IOC contextual attribute. Macro precision is the unweighted average of per-class precision within the attribute, based on the same value-class definition as above.

Averages are computed over all IOC contextual attributes combined across both extractors, and the reported metric values are conditioned on each LLM’s extracted IOCs.

The variation in results across LLMs reflects the interplay of several factors, including differences in their capacity to detect, link, and interpret cues, the extent of permitted inference, adherence to instructions and instruction–model fit, and characteristics of the input CTI reports themselves. As discussed earlier, CTI reports vary widely along multiple dimensions relevant to LLM‑driven information extraction, such as evidence strength, terminology, and format.

In practice, selecting an LLM for CTI information extraction and integrating its outputs into downstream applications requires setting accuracy and precision thresholds and weighing operational factors such as latency, all aligned with the requirements of the intended application. For example, fully automated mission‑critical applications warrant stricter thresholds than exploratory uses. Thresholds may be defined globally across all outputs and, where relevant, per output category.

Value Assignment Abstention

In our extraction pipeline, the value class None provides an explicit abstention option for value assignment, allowing the LLM to assign None to an IOC contextual attribute when evidence is insufficient, rather than outputting a concrete value. Since CTI reporting often provides only partial or implied cues supporting a definitive assignment, and at times no relevant cues at all, an abstention option is important: without it, the LLM would have to commit to an output despite insufficient evidence, inflating false positives and undermining trust in the outputs. By enabling abstention, a value such as None reduces incorrect assignments, communicates uncertainty, and allows downstream consumers to defer, escalate, or exclude that data point.

The abstention option requires careful consideration because it trades correctness, including accuracy and precision, against coverage. For instance, a lenient inference policy, which accepts weak evidence and broader contextual cues, reduces abstention and increases coverage but raises the risk of speculative assignments. In contrast, a strict policy that requires strong evidence and limits inference increases abstention and improves correctness but may suppress recoverable information.

Building on the accuracy and precision evaluation above, this section focuses on the LLMs’ abstention behavior, specifically their assignments of None. We report two error rates:

• False Discovery Rate (FDR): the proportion of None assignments that were unwarranted (the LLM assigned None while the ground truth specified a non-None value), indicating excessive conservatism; and
• False Negative Rate (FNR), the proportion of instances that should have abstained but did not (the LLM assigned a non-None value while the ground truth was None), indicating a tendency to speculate.

The observed variation in value assignment abstention across LLMs highlights the importance of evaluating this aspect of model behavior. Evaluation of abstention tendencies guides LLM selection and configuration, helps define acceptable ranges for abstention and speculative assignments appropriate to the use case, and informs the choice of operating settings that balance correctness and coverage, such as evidence criteria and the extent of permitted inference. Abstention behavior requires ongoing monitoring as input data changes over time to keep its frequency and speculation rates within target ranges for downstream applications.

Evaluation | LLM Ensemblies

Ensembling multiple LLMs can improve extraction correctness and stability by offsetting model‑specific limitations. Examples include majority voting, where the most frequent prediction across LLMs is selected, and judge‑based arbitration, in which one LLM reconciles conflicting outputs.

Effective ensembles balance operational compatibility, such as comparable inference latency and extraction performance, with statistical diversity. For example, when individual LLM accuracies differ substantially, an ensemble may provide little or no improvement. Under such conditions, a majority‑voting configuration with unweighted aggregation can even reduce overall accuracy, whereas weighted schemes that assign greater weight to more accurate models tend to converge toward the output of the strongest single model.

The potential benefit of any ensemble ultimately depends on the diversity of predictions and errors among its members. If models fail in similar ways, aggregation merely amplifies shared weaknesses, whereas if their errors differ or their predictions diverge, ensembling can provide more reliable and accurate results by combining complementary reasoning.

To illustrate this concept, the chart below reports the phi (φ) error correlation coefficient and the disagreement rate, calculated from the extraction outputs of GPT‑4.1 and Claude Sonnet 4.5 when operating within the Infrastructure and Executables Extractor. The error correlation coefficient measures the extent to which the two LLMs make the same mistakes, while the disagreement rate captures how often their predictions diverge on the same extraction field. Low error correlation combined with moderate disagreement indicates complementary reasoning and strong ensemble potential. In contrast, high error correlation and low disagreement suggest that the LLMs fail in similar ways, limiting the benefit of aggregation.

Both metrics were calculated on the same set of IOCs and corresponding contextual attributes for which the two LLMs produced predictions. The analysis focuses on a subset of IOC contextual attributes chosen to illustrate how error diversity manifests across attributes that differ in value format (categorical and open-text) and reasoning demands, ranging from typically localized factual attributes (is_compromised and injection) to contextual and functional (usage, execution_form, and attack_stage) and explicitly stated attributes (filepath).

The results show variable ensemble potential across attributes, with predictions for some attributes, such as attack_stage, showing more complementary behavior between models, while others, such as usage, display strong coupling in their errors. This heterogeneity suggests that ensemble benefit is influenced by the interaction between the reasoning demands of each attribute and the way individual models respond to those demands in their predictions.

LLM ensemble configurations for CTI information extraction should therefore be evaluated on a task‑specific basis, such as per IOC contextual attribute in this study, rather than applied uniformly across all extraction tasks. Selective, empirically guided use of ensembling provides a more targeted path to maximizing its contribution to overall system performance.

Evaluation | Playbook and Knowledge Graph Assembly

In this section, we evaluate each LLM within the Playbook Extractor, focusing on its ability to construct connected, semantically coherent representations of adversary behavior described in CTI reports. Specifically, we examine how effectively each LLM instantiates ThreatActor, Playbook, and Step data model entities, and links them through sequencing, MITRE ATT&CK mappings, and IOC associations with threat actor actions to form a unified knowledge graph. In practical terms, this evaluation measures each model’s capacity to reconstruct the full sequence of threat actor actions within an adversary operation, ensuring that the resulting representations are internally consistent, chronologically coherent, and semantically valid.

The analysis is based on 17 individual metrics, each expressed as a ratio between 0 and 1 representing the proportion of structural or semantic elements (such as data model entities, links and their typed relationships, MITRE ATT&CK mappings, and IOC associations) that satisfy defined validation rules or external references, out of all instances evaluated for the respective metric. Here, rules refer to internal consistency conditions that shape a valid Playbook or graph structure (for example, acyclic sequences, reachability of Step entities, absence of orphaned Step entities), whereas references denote external knowledge sources used to check semantic accuracy (for example, a list of valid MITRE ATT&CK tactics and techniques and their parent-child relationships).

We consolidate these individual ratios into four aggregate categories, where each category’s value is the mean of its constituent ratios, capturing a distinct dimension of reconstruction quality:

• Structural Integrity: Assesses how coherent and complete each reconstructed Playbook and the resulting knowledge graph is, for example how many Playbook instances are loop‑free, fully connected, and internally consistent, as well as the extent to which atomic IOCs and their contextual attributes are linked to Step entities.
• ATT&CK Mapping Validity: Measures the correctness of MITRE ATT&CK mappings and hierarchies, including the rate of valid tactic, technique, and sub‑technique identifiers and the proportion of correctly formed parent‑child relationships.
• Complexity and Semantic Diversity: Reflects how detailed and varied the reconstructed threat actor actions are, considering both the diversity of the captured ATT&CK tactics and techniques and the level of procedural detail expressed through the number of Step entities within each Playbook.
• IOC Integration Density: Evaluates how thoroughly threat actor actions are associated with specific atomic IOCs and their contextual attributes, expressed through the average number of atomic IOCs linked per Step entity.

For the Structural Integrity and ATT&CK Mapping Validity categories, higher values indicate greater structural and semantic correctness, reaching 1.0 for fully valid results. The Complexity and Semantic Diversity, and IOC Integration Density, categories are based on normalized ratios that asymptotically approach 1.0 and provide relative measurements of how detailed, varied, and tightly interconnected each model’s reconstructions are. Building on the category aggregates, we calculate an overall Correctness Score as the mean of the Structural Integrity and ATT&CK Mapping Validity category scores, providing a concise and aggregate measure of structural and semantic correctness.

The chart below summarizes the category scores and the corresponding Correctness Score for each evaluated LLM.

Despite relatively strong performance in transforming CTI report content into interlinked representations, the LLMs’ use of generative reasoning for extraction, combined with ambiguity and uneven detail in many reports, can introduce inconsistencies or omissions in the reconstructed structures, reducing overall coherence. These issues can affect how downstream applications traverse, correlate, and reason over the extracted information, and they should be explicitly accounted for in the design and integration of analytical workflows that consume these reconstructions. For example, implementations may prioritize the mission‑critical portions of the reconstructed structure, such as subgraphs whose relationships are key to the intended use case and must remain accurately captured to support consistent traversal and analysis, and apply additional assurance measures. Such measures include, for example, refined prompt design with strict generation guardrails or automated consistency checks.

Conclusions

LLMs can effectively automate information extraction from CTI reports, delivering substantial speed gains over manual processing. However, these reports vary widely in structure, terminology, and level of evidentiary detail, and the contextual cues needed to support LLM inference for a given extraction task may be implicit, inconsistent, or absent.

Beyond report variability, extraction outcomes also depend on the model’s reasoning capacity to connect contextual cues and on the applied inference policy applied. Together, these factors can lead to inaccuracies and coverage gaps.

In practice, achieving reliable results requires deliberate planning, evaluation, and continuous refinement. Operationalizing LLM-based CTI information extraction means setting clear objectives, defining standards for evidence and output quality, and investing in robust evaluation processes supported by representative ground truth data, all aligned with the intended application. Effective deployment depends as much on well-defined processes as on model choice. These processes involve balancing factors such as accuracy, coverage, and latency to meet operational requirements, as well as building safeguards and contingencies for mission-critical downstream applications.

Looking ahead, future model generations with stronger reasoning, better long‑range attention and salience, and more consistent adherence to extraction constraints than current models can raise baseline extraction correctness and coverage. For CTI and cyber defense, this means more accurate and complete structured intelligence, produced at scale from diverse narratives, which, in turn, supports more reliable correlation and prioritization, strengthens detection and response, and enables broader reuse across tools and teams. We remain committed to sharing insights that support CTI teams and cyber defense organizations in integrating AI capabilities within their workflows.

Executive Summary

• A joint research project between SentinelLABS and Censys reveals that open-source AI deployment has created an unmanaged, publicly accessible layer of AI compute infrastructure spanning 175,000 hosts worldwide, operating outside the guardrails and monitoring systems that platform providers implement by default.
• Over 293 days of scanning, we identified 7.23 million observations across 130 countries, with a persistent core of 23,000 hosts generating the majority of activity.
• Nearly half of observed hosts are configured with tool-calling capabilities that enable them to execute code, access APIs, and interact with external systems demonstrating the increasing implementation of LLMs into larger system processes.
• Hosts span cloud and residential networks globally, but overwhelmingly run the same handful of AI models in identical formats, creating a brittle monoculture.
• The residential nature of much of the infrastructure complicates traditional governance and requires new approaches that distinguish between managed cloud deployments and distributed edge infrastructure.

Background

Ollama is an open-source framework that enables users to run large language models locally on their own hardware. By design, the service binds to localhost at 127.0.0.1:11434, making instances accessible only from the host machine. However, exposing Ollama to the public internet requires only a single configuration change: setting the service to bind to 0.0.0.0 or a public interface. At scale, these individual deployment decisions aggregate into a measurable public surface.

Over the past year, as open-weight models have proliferated and local deployment frameworks have matured, we observed growing discussion in security communities about the implications of this trend. Unlike platform-hosted LLM services with centralized monitoring, access controls, and abuse prevention mechanisms, self-hosted instances operate outside emerging AI governance boundaries. To understand the scope and characteristics of this emerging ecosystem, SentinelLABS partnered with Censys to scan and map internet-reachable Ollama deployments.

Our research aimed to answer several questions: How large is the public exposure? Where do these hosts reside? What models and capabilities do they run? And critically, what are the security implications of a distributed, unmanaged layer of AI compute infrastructure?

The Exposed Ecosystem | Scale and Structure

Our scanning infrastructure recorded 7.23 million observations from 175,108 unique Ollama hosts across 130 countries and 4,032 autonomous system numbers (ASNs). The raw numbers suggest a substantial public surface, but the distribution of activity reveals a more nuanced picture.

The ecosystem is bimodal. A large layer of transient hosts sits atop a smaller, persistent backbone that accounts for the majority of observable activity. These transient hosts appear briefly and then disappear. Hosts that appear in more than 100 observations represent just 13% of the unique host population, yet they generate nearly 76% of all observations. Conversely, hosts observed exactly once constitute 36% of unique hosts but contribute less than 1% of total observations.

This persistence skew shapes the rest of our analysis. It’s why model rankings stay stable even as the host population grows, why the host counts look residential while the always-on endpoints behave more like cloud services, and why most of the security risk sits in a smaller subset of exposed systems.

Regardless of this skew, persistent hosts that remain reachable across multiple scans comprise the backbone of our data. This is where capability, exposure, and operational value converge. These are systems that provide ongoing utility to their operators and, by extension, represent the most attractive and accessible targets for adversaries.

Infrastructure Footprint and Attribution Challenges

The infrastructure distribution challenges assumptions about where AI compute resides. When classified by ASN type, fixed-access telecom networks, which include consumer ISPs, constitute the single largest category at 56% of hosts by count. However, when the same data is grouped into broader infrastructure tiers, exposure divides almost evenly: Hyperscalers account for 32% of hosts, and Telecom/Residential networks account for another 32%.

This apparent contradiction reflects a classification and attribution challenge inherent in internet scanning. Both views are accurate, and together they indicate that public Ollama exposure spans a mixed environment. Access networks, independent VPS providers, and major cloud platforms all serve as durable habitats for open-weight LLM deployment.

Operational characteristics vary by tier. Indie Cloud/VPS environments show high average persistence and elevated “running share,” which measures the proportion of hosts actively serving models at scan time. This is consistent with endpoints that provide stable, ongoing service. Telecom/Residential hosts, by contrast, report larger average model inventories but lower running share, suggesting machines that accumulate models over time but operate intermittently.

Geographic distribution also reveals concentration patterns. In the United States, Virginia alone accounts for 18% of U.S. hosts, likely reflecting the density of cloud infrastructure in US-EAST. In China, concentration is even tighter: Beijing accounts for 30% of Chinese hosts, with Shanghai and Guangdong contributing an additional 21% combined. These patterns suggest that observable open-source AI capability concentrates at infrastructure hubs rather than distributing uniformly.

A significant portion of the infrastructure footprint, however, resists clean attribution. Depending on the classification method, 16% of tier labels and 19% of ASN-type classifications returned null values in our scans. This attribution gap reflects a governance reality. Security teams and enforcement authorities can observe activity, but they often cannot identify the responsible party. Traditional mechanisms that rely on clear ownership chains and abuse contact points become less effective when nearly one-fifth of the infrastructure is anonymous.

Model Adoption and Hardware Constraints

Although nothing is truly uniform on the internet, in our data we observe a distinct trend. Host placement is decentralized, but model adoption is concentrated. Lineage rankings are exceptionally stable across multiple weighting schemes. Across observations, unique hosts, and host-days, the same three families occupy the same positions with zero rank volatility: Llama at #1, Qwen2 at #2, and Gemma2 at #3. This stability indicates broad, repeated use of shared model lineages rather than a fragmented, experiment-heavy deployment pattern.

Portfolio behavior reveals a shift toward multi-model deployments. The average number of models per observation rose from 3 in March to 4 by September-December. The most common configuration remains modest at 2-3 models, accounting for 41% of hosts, but a small minority of “public library” hosts carry 20 or more models. These represent only 1.46% of hosts but disproportionately drive model-instance volume and family diversity.

Co-deployment patterns suggest operational logic beyond simple experimentation. The most prominent multi-family pairing, llama + qwen2, appears on 40,694 hosts, representing 52% of multi-family deployments. This consistency suggests operators maintain portfolios for comparison, redundancy, or workload segmentation rather than committing to a single lineage.

Hardware constraints express themselves clearly in quantization preferences and parameter-size distributions as well. The deployment regime converges strongly on 4-bit compression. The specific format Q4_K_M appears on 48% of hosts, and 4-bit formats total 72% of all observed quantizations compared to just 19% for 16-bit. This convergence is not confined to a single infrastructure niche. Q4_K_M ranks #1 across Academic, Hyperscaler, Indie VPS, and Telecom/Residential tiers.

Parameter sizes cluster in the mid-range. The 8-14B band is most prevalent at 26% of hosts, with 1-3B and 4-7B bands close behind. Together, these patterns reflect the practical economics of running inference on commodity hardware: models must be small enough to fit in available VRAM and memory bandwidth but also be capable enough for practical work.

This ecosystem-wide convergence on specific packaging regimes creates both portability and fragility. The same compression choices that enable models to run across diverse hardware environments also create a monoculture. A vulnerability in how specific quantized models handle tokens could affect a substantial portion of the exposed ecosystem simultaneously rather than manifesting as isolated incidents. This risk is particularly acute for widely deployed formats like Q4_K_M.

Capability Surface | Tools, Modalities, and Intent Signals

The persistent backbone is configured for action. Over 48% of observed hosts advertise tool-calling capabilities via their API endpoints. When queried, hosts return capability metadata indicating which operations they support. The specific combination of [completion, tools] indicates a host that can both generate text and execute functions. This configuration appears on 38% of hosts, indicating systems wired to interface with external software, APIs, or file systems.

Modality support extends beyond text. Vision capabilities appear on 22% of hosts, enabling image understanding and creating vectors for indirect prompt injection via images or documents. “Thinking” models, which are optimized for multi-step reasoning and chain-of-thought processing, appear on 26% of hosts. When paired with tool-calling capabilities, reasoning capacity acts as a planning layer that can decompose complex tasks into sequential operations.

System prompt analysis surfaced a subset of deployments with explicit intent signals. We identified at least 201 hosts running standardized “uncensored” prompt templates that explicitly remove safety guardrails. This count represents a lower bound; our methodology captured only prompts visible via API responses and the presence of standardized “guard-off” configurations indicates a repeatable pattern rather than isolated experimentation.

A subset of 5,000 hosts demonstrates both high capability and high availability, showing 87% average uptime while actively running an average of 1.8 models. This combination of persistence, tool-enablement, and consistent availability suggests endpoints that provide ongoing operational value and, from an adversary perspective, represent stable, accessible compute resources.

Security Implications

The exposed Ollama ecosystem presents several threat vectors that differ from risks associated with platform-hosted LLM services.

Resource Hijacking

The persistent backbone represents a new network layer of compute infrastructure that can be accessed without authentication, usage monitoring, or billing controls. Frontier LLM providers have reported that criminal organizations and state-sponsored actors leverage their platforms for spam campaigns, phishing, disinformation networks, and network exploitation. These providers deploy dedicated security and fraud teams, implement rate limiting, and maintain abuse detection systems.

In contrast, the exposed Ollama backbone offers adversaries distributed compute resources with minimal centralized oversight. An attacker can direct malicious workloads to these hosts at zero marginal cost. The victim pays the electricity bill and infrastructure costs while the attacker receives the generated output. For operations requiring volume, such as spam generation, phishing content creation, or disinformation campaigns, this represents a substantial operational advantage.

Excessive Agency

Tool-calling capabilities fundamentally alter the threat model. A text-generation endpoint can produce harmful content, but a tool-enabled endpoint can execute privileged operations. When combined with insufficient authentication and network exposure, this creates what we assess to be the highest-severity risk in the ecosystem.

Prompt injection becomes an increasingly important threat vector as LLM enabled systems  are provided increased agency. This technique manipulates LLM behavior through crafted inputs. An attacker no longer needs to breach a file server or database; they can prompt an exposed Retrieval-Augmented Generation instance with benign-sounding requests: “Summarize the project roadmap,” “List the configuration files in the documentation,” or “What API keys are mentioned in the codebase?” A model designed to be helpful and lacking authentication or safety mechanisms, will comply with these requests if its retrieval scope includes the targeted information.

We observed configurations consistent with retrieval workflows, including “chat + embeddings” pairings that suggest RAG deployments. When these systems are internet-reachable and lack access controls, they represent a direct path from external prompt to internal data.

Identity Laundering and Proxy Abuse

A significant portion of the exposed ecosystem resides on residential and telecom networks. These IP addresses are generally trusted by internet services as originating from human users rather than bots or automated systems. This creates an opportunity for sophisticated attackers to launder malicious traffic through victim infrastructure.

With vision capabilities present on 22% of hosts, indirect prompt injection via images becomes viable at scale. An attacker can embed malicious instructions in an image file and, if a vision-capable Ollama instance processes that image, trigger unintended behavior. When combined with tool-calling capabilities on a residential IP, this enables attacks where malicious traffic appears to originate from a legitimate household, bypassing standard bot management and IP reputation defenses.

Concentration Risk

The ecosystem’s convergence on specific model families and quantization formats creates systemic fragility. If a vulnerability is discovered in how a particular quantized model architecture processes certain token sequences, defenders would face not isolated incidents but a synchronized, ecosystem-wide exposure. Software monocultures have historically amplified the impact of vulnerabilities. When a single implementation error affects a large percentage of deployed systems, the blast radius expands accordingly. The exposed Ollama ecosystem exhibits this pattern: nearly half of all observed hosts run the same quantization format, and the top three model families dominate across all measurement methods.

Governance Gaps

Effective cybersecurity incident response relies on clear attribution: identifying the owner of compromised infrastructure, issuing takedown notices, and escalating through established abuse reporting channels. Even where attribution succeeds, enforcement mechanisms assume centralized control points. In cloud environments, providers can disable instances, revoke credentials, or implement network-level controls. In residential and small VPS environments, these levers often do not exist. An Ollama instance running in a home network or on a low-cost VPS may be accessible to adversaries but unreachable by security teams lacking contractual or legal authority.

Open Weights and the Governance Inversion

The exposed Ollama ecosystem forces a distinction that “open” rhetoric often blurs: distribution is decentralized, but dependency is centralized. On the ground, public instances span thousands of networks and operator types, with no single provider controlling where they live or how they’re configured, yet at the model-supply layer, the ecosystem repeatedly converges on the same few options. Lineage choice, parameter size, and quantization format determine what is actually runnable or exploitable.

This creates what we characterize as a governance inversion. Accountability diffuses downward into thousands of home networks and server closets, while functional dependency concentrates upward into a handful of model lineages released by a small number of labs. Traditional governance frameworks assume the opposite: centralized deployment with diffuse upstream supply.

In platform-hosted AI services, governance flows through service boundaries.This includes all too familiar terms of use, API rate limits, content filtering, telemetry, and incident response capacity. Open-weight models operate differently. Providers can monitor usage patterns, detect abuse, and terminate access for policy violations including use in state-sponsored campaigns. In artifact-distributed models, these mechanisms largely do not exist. Weights behave like software artifacts: copyable, forkable, quantized into new formats, retrainable and embedded into stacks the releasing lab will never observe.

Our data makes the artifact model difficult to ignore. Infrastructure placement is widely scattered, yet operational behavior and capability repeatedly trace back to upstream release decisions. When a new model family achieves portability across commodity hardware and gains adoption, that release decision gets amplified through distributed deployment at a pace that outstrips existing governance timelines.

This dynamic does not mean open weights are inherently problematic – the same characteristics that create governance challenges also enable research, innovation, and deployment flexibility that platform-hosted services cannot match. Rather, it suggests that governance mechanisms designed for centralized platforms require adaptation to this new risk environment. Post-release monitoring, vulnerability disclosure processes, and mechanisms for coordinating responses to misuse at scale become critical when frontier capability is produced by a few labs but deployed everywhere.

Conclusion

The exposed Ollama ecosystem represents what we assess to be the early formation of a public compute substrate: a layer of AI infrastructure that is widely distributed, unevenly managed, and only partially attributable, yet persistent enough in specific tiers and locations to constitute a measurable phenomenon.

The ecosystem is structurally paradoxical. It is resilient in its spread across thousands of networks and jurisdictions, making it impossible to “turn off” through centralized action, yet it is fragile in its dependency, relying on a narrow set of upstream model lineages and packaging formats. A single widespread vulnerability or adversarial technique optimized for the dominant configurations could affect a substantial portion of the exposed surface.

Security risk concentrates in the persistent backbone of hosts that remain consistently reachable, tool-enabled, and often lacking authentication. These systems require different governance approaches depending on infrastructure tier: traditional controls for cloud deployments, but sanitation mechanisms for residential networks where contractual leverage does not exist.

For defenders, the key takeaway is that LLMs are increasingly deployed to the edge to translate instructions into actions. As such, they must be treated with the same authentication, monitoring, and network controls as other externally accessible infrastructure.

In this talk, Phobos Group’s Dan Tentler evolves his previous work on hotel room security by demonstrating a fully portable security system built on Home Assistant, Z-Wave devices, CO2 sensors, and millimeter wave radar. What began as basic physical security measures has transformed into a tactical deployment platform capable of detecting human presence through walls, triggering automated alerts, and providing comprehensive situational awareness in temporary accommodations.

Dan walks through the technical fundamentals of each component, explaining how mmWave radar units can detect movement and presence in neighboring rooms or hallways, how CO2 sensors reveal occupancy patterns, and how Home Assistant ties everything together into an automation framework. The system can send alerts, capture images, or trigger any action Home Assistant supports, all deployed and configured rapidly in unfamiliar environments.

The presentation covers real-world use cases that demonstrate the system’s capabilities beyond traditional hotel rooms. For security professionals, researchers, and anyone concerned with physical security while traveling, this talk reveals how consumer automation technology can be repurposed into a sophisticated portable security platform.

About the Author

Dan Tentler is the Executive Founder and CTO of Phobos Group, a boutique information security services and products company. Having been on both red and blue teams, Dan brings a wealth of defensive and adversarial knowledge to the security landscape. Dan has spent time at Twitter, British Telecom, Websense, Anonymizer, Intuit and Sempra Energy and has a strong background in systems, networking, architecture and wireless networks.

About LABScon

This presentation was featured live at LABScon 2025, an immersive 3-day conference bringing together the world’s top cybersecurity minds, hosted by SentinelOne’s research arm, SentinelLABS.

Keep up with all the latest on LABScon here.

Executive Summary

• SentinelLABS’ analysis of benchmarks for  LLM in cybersecurity, including those published by major players such as Microsoft and Meta, found that none measure what actually matters for defenders.
• Most LLM benchmarks test narrow tasks, but these map poorly to security workflows, which are typically continuous, collaborative, and frequently disrupted by unexpected changes.
• Models that excel at coding and math provide minimal direct gains on security tasks, indicating that general LLM capabilities do not readily translate to analyst-level thinking.
• All of today’s benchmarks use LLMs to evaluate other LLMs, often using the same vendor’s models for both, creating a closed loop that is more susceptible to gaming, and difficult to trust.
• As frontier labs push defenders to rely on models to automate security operations, the importance of benchmarks will increase drastically as the main mechanism to evaluate whether the capabilities of the models match the vendor’s claims.

For security teams, AI promised to write secure code, identify and patch vulnerabilities, and replace monotonous security operations tasks. Its key value proposition was raising costs for adversaries while lowering them for defenders.

To evaluate whether Large Language Models were both performant and reliable enough to be deployed into the enterprise, a wave of new benchmarks were created. In 2023, these early benchmarks largely comprised multiple-choice exams over clean text, which produced clean and reproducible metrics for performance. However, as the models improved they outgrew the early tests: scores across models began to converge at the top of the scale as the benchmarks became increasingly “saturated”, and the tests themselves ceased telling anything meaningful.

As the industry has boomed over the past few years, benchmarking has become a way to distinguish new models from older ones. Developing a benchmark that shows how a smaller model outperforms a larger one released from a frontier AI lab is a billion-dollar industry, and now every new model launches with a menagerie of charts with bold claims. +3.7 on SomeBench-v2, SOTA on ObscureQA-XL, or 99th percentile on an-exam-no-one-had-heard-of-last-week. The subtext here is simple: look at the bold numbers, be impressed, and please join our seed round!

Inside this swamp of scores and claims, security teams are somehow meant to conclude that a system is safe enough to trust with an organization’s business, its users, and maybe even its critical infrastructure. However, a careful read through the arxiv benchmark firehose reveals a hard-to-miss pattern: We have more benchmarks than ever, and somehow we are still not measuring what actually matters for defenders.

So what do security benchmarks actually measure? And how well does this approach map to real security work?

In this post, we review four popular LLM benchmarking evaluations: Microsoft’s ExCyTIn-Bench, Meta’s CyberSOCEval and CyberSecEval 3, and Rochester Institute’s CTIBench. We explore what we think these benchmarks get right and where we believe they fall short.

What Current Benchmarks Actually Measure

ExCyTIn-Bench | Realistic Logs in a Microsoft Snow Globe

ExCyTIn-Bench was the cleanest example of an “agentic” Security Operations benchmark that we reviewed. It drops LLM agents into a MySQL instance that mirrors a realistic Microsoft Azure tenant. They provide 57 Sentinel-style tables, 8 distinct multi-stage attacks, and a unified log stream spanning 44 days of activity.

Each question posed to the LLM agent is anchored to an incident graph path. This means that the agent must discover the schema, issue SQL queries, pivot across entities, and eventually answer the question. Rewards for the agent are path-aware, meaning that full credit is assigned for the right answer, but the agent could also earn partial credit for each correct intermediate step that it took.

The headline result is telling:

“Our comprehensive experiments with different models confirm the difficulty of the task: with the base setting, the average reward across all evaluated models is 0.249, and the best achieved is 0.368…” (arxiv)

Microsoft’s  ExCyTIn benchmark demonstrates that LLMs struggle to plan multi-hop investigations over realistic, heterogeneous logs.

This is an important finding – especially for those who are concerned with how LLMs work in real world scenarios. Moreover, all of this takes place in a Microsoft snow globe: one fictional Azure tenant, eight well-studied, canned attacks and clean tables and curated detection logic for the agent to work with. Although the realistic agent setup is a massive improvement over trivia-style Multiple Choice Question (MCQ) benchmarks, it is not the daily chaos of real security operations.

CyberSOCEval | Defender Tasks Turned into Exams

CyberSOCEval is part of Meta’s CyberSecEval 4 and deliberately picks two tasks defenders care about: malware analysis over real sandbox detonation logs and threat Intelligence reasoning over 45 CTI reports. The authors open with a statement we very much agree with:

“This lack of informed evaluation has significant implications for both AI developers and those seeking to apply LLMs to SOC automation. Without a clear understanding of how LLMs perform in real-world security scenarios, AI system developers lack a north star to guide their development efforts, and users are left without a reliable way to select the most effective models.” (arxiv)

To evaluate these tasks, the benchmark frames them as multi-answer multiple-choice questions and incorporates analytically computed random baselines and confidence intervals. This setup gives clean, statistically grounded comparisons between models and reduces complex workflows into simplified questions. Researchers found that the models perform far above random, but also far from solved.

In the malware analysis trial, they score exact-match accuracy in the teens to high-20s percentage range versus a random baseline around 0.63%. For threat-intel reasoning, models land in the ~43 to 53% accuracy band versus ~1.7% random.

In other words, the models are clearly extracting meaningful signals from real logs and CTI reports. However, the models also are failing to correctly answer most of the malware questions and roughly half of the threat intelligence questions.

These findings suggest that for any system aimed at automating SOC workflows, model performance should be evaluated as assistive rather than autonomous.

Crucially, they find that test-time “reasoning” models don’t get the same uplift they see in math/coding:

“We also find that reasoning models leveraging test time scaling do not achieve the boost they do in areas like coding and math, suggesting that these models have not been trained to reason about cybersecurity analysis…” (arxiv)

That’s a big deal, and it’s evidence that you don’t get generalized security reasoning for free just by cranking up “thinking steps”.

Meta’s CyberSOCEval falls short because it compresses two complex domains into MCQ exams. There is no notion of triaging multiple alerts or asking follow-up questions or hunting down log sources. In real life, analysts need to decide when to stop and escalate or switch paths.

In the end, while the CyberSOCEval is a clean and statistically sound probe of model performance on a set of highly-specific sub-tasks, it is far from a representation of how SOC workflows should be modeled.

CTIBench | CTI as a Certification Exam

CTIBench is a benchmark task suite introduced by researchers at Rochester Institute of Technology to evaluate how well LLMs operate in the field of Cyber Threat Intelligence. Unlike general purpose benchmarks, which focus on high-level domain knowledge, CTIBench grounds tasks in the practical workflows of information security analysts. Like other benchmarks we examined it performs this analysis as an MCQ exam.

“While existing benchmarks provide general evaluations of LLMs, there are no benchmarks that address the practical and applied aspects of CTI-specific tasks.” (NeurIPS Papers)

CTIbench draws on well-known security standards and real-world threat reports, then turns them into five kinds of tasks:

• basic multiple-choice questions about threat-intelligence knowledge
• mapping software vulnerabilities to their underlying weaknesses
• estimating how serious a vulnerability is
• pulling out the specific attacker techniques described in a report
• guessing which threat group or malware family is responsible.

The data is mostly from 2024, so it’s newer than what most models were trained on, and each task is graded with a simple “how close is this to the expert answer?” style score that fits the kind of prediction being made.

On paper, this looks close to the work CTI teams care about: mapping vulnerabilities to weaknesses, assigning severity, mapping behaviors to techniques, and tying reports back to actors.

In practice, though, the way those tasks are operationalized keeps the benchmark in the frame of a certification exam. Each task is cast as a single-shot question with a fixed ground-truth label, answered in isolation with a zero-shot prompt. There is no notion of long-running cases, heterogeneous and conflicting evidence, evolving intelligence, or the need to cross-check and revise hypotheses over time.

CTIBench is yet another MCQ, an excellent exam if you want to know, “Can this model answer CTI exam questions and do basic mapping/annotation?” It says less about whether an LLM can do the messy work that actually creates value: normalizing overlapping feeds, enriching and de-duplicating entities in a shared knowledge graph, negotiating severity and investment decisions with stakeholders, or challenging threat attributions that don’t fit an organization’s historical data.

CyberSecEval 3 | Policy Framing Without Operational Closure

CyberSecEval 3, also from Meta, is not a SOC benchmark so much as a risk map. The authors carve the space into eight risks, grouped into two buckets: harms to third parties i.e., offensive capabilities and harms to application developers and end users such as misuse, vulnerabilities, or data leakage. The frame of this eval is the current regulatory conversation between governments and standards bodies about unacceptable model risk, so the suite is understandably organized around “where could this go wrong?” rather than “how much better does this make my security operations?”

The benchmark’s coverage tracks almost perfectly with the concerns of policymakers and safety orgs. On the offensive side, CyberSecEval 3 looks at automated spear-phishing against LLM-simulated victims, uplift for human attackers solving Hack-The-Box style CTF challenges, fully autonomous offensive operations in a small cyber range, and synthetic exploit-generation tasks over toy programs and CTF snippets. On the application side, it probes prompt injection, insecure code generation in both autocomplete and instruction modes, abuse of attached code interpreters, and the model’s willingness to help with cyberattacks mapped to ATT&CK stages.

The findings across these areas are very broad. Llama3 is described as capable of “moderately persuasive” spear-phishing, roughly on par with other SOTA models when judged against simulated victims. In the CTF study, Llama3 405B gives novice participants a noticeable bump in completed phases and slightly faster progress, but the authors stress that the effect is not statistically robust.

The fully autonomous agent can handle basic reconnaissance in the lab environment, but fails to achieve reliable exploitation or persistence. On the application-risk side, all tested models suggest insecure code at non-trivial rates, prompt injection succeeds a significant fraction of the time, and models will sometimes execute malicious code or provide help with cyberattacks. Meta stresses that its own guardrails reduce these risks on the benchmark distributions.

CyberSecEval 3 may have some value for those working in policy and governance. None of the eight risks are defined in terms of operational metrics such as detection coverage, time to triage, containment, or vulnerability closure rates. The CTF experiment comes closest to demonstrating something about real-world value, but it is still an artificial one-hour lab on pre-selected targets. Moreover, this experiment is expensive and not reproducible at scale.

There are glimmers of this in the paper, and CyberSecEval3 remains a strong contribution to AI security understanding and governance, but a weak instrument for deciding whether to deploy a model as a copilot for live operations.

Benchmarks are Measuring Tasks, not Workflows

All of these benchmarks share a common blind spot: they treat security as a collection of isolated questions rather than as an ongoing workflow.

Real teams work through queues of alerts, pivot between partially related incidents, and coordinate across levels of seniority. They make judgment calls under time pressure and incomplete telemetry. Closing out a single alert or scoring 90% on a multiple choice test is not the goal of a security team. The goal is reducing the underlying risk to the business, and  this means knowing the right questions to ask in the first place.

ExCyTIn-Bench comes closest to acknowledging this reality. Agents interact with an environment over multiple turns and earn rewards for intermediate progress. Yet even here, the fundamental unit of evaluation is still a question: “What is the correct answer to this prompt?” The system is not asked to “run this incident to ground” or evaluate different environments or logging sources that may be included in an incident response. CyberSOCEval and CTIBench compress even richer workflows into single multiple-choice interactions.

Methodologically, this means none of these benchmarks are measuring outcomes that define security performance. Metrics such as time-to-detect, time-to-contain, and mean time to remediate are absent. We are measuring how models behave when the important context has already been carefully prepared and handed to them, not how they behave when dropped into a live incident where they must decide what to look at, what to ignore, and when to ask for help.

Until we are ready to benchmark at the workflow level, we should understand that high accuracy on multiple-choice security questions and smooth reward curves are not stand-ins for operational uplift. In information security, the bar must be higher than passing an exam.

MCQs and Static QA are Overused Crutches

Multiple-choice questions are attractive for understandable reasons. They are easy to score at scale. They support clean random baselines and confidence intervals and they fit nicely into leaderboards and slide decks.

The downside is that this format quietly bakes in assumptions that do not hold in practice. For any given scenario, the benchmark assumes someone has already asked the right question. There is no space for challenging the premise of that question, reframing the problem, or building and revising a plan. All of the relevant evidence has already been selected and pre-packaged for the analyst. In that setting, the model’s job is essentially to compress and restate context, not to decide what to investigate or how to prioritize effort. Wrong or partially correct answers carry no real cost.

This is the inverse of real SOC and CTI work where the hardest part is deciding what questions to ask, what data to pull, and what to ignore. That judgment ability is usually earned over years of experience or deliberate training, If we want to know whether models will actually help in our workflows, we need evaluations where asking for more data has a cost, ignoring critical signals is penalized, and “I don’t know, let me check” is a legitimate and sometimes optimal response.

Statistical Hygiene is Still Uneven

To their credit, some of these efforts take statistics seriously. CyberSOCEval reports confidence intervals and uses bootstrap analysis to reason about power and minimum detectable effect sizes. CTIBench distinguishes between pre- and post-cutoff datasets and examines performance drift. CyberSecEval 3 uses survival analysis and appropriate hypothesis tests in its human-subject CTF study to show an unexpected lack of statistically significant uplift from an LLM copilot.

Across the board, however, there are still gaps. Many results come from single-seed, temperature-zero runs with no variance reported. ExCyTIn-Bench, for instance, reports an average reward of 0.249 and a best of 0.368, but provides no confidence intervals or sensitivity analysis. Contamination is rarely addressed systematically, even though all four benchmarks draw on well-known corpora that almost certainly overlap with model training data. Heavy dependence on a single LLM judge, often from the same vendor as the model being evaluated, compounds these issues.

The consequence is that headline numbers can look precise while being fragile under small changes in prompts, sampling parameters, or judge models. If we expect these benchmarks to inform real governance and deployment decisions, variance, contamination checks, and judge robustness should be baseline, check-box requirements.

Using LLMs to Evaluate LLMs Is Everywhere, and Rarely Questioned

Every benchmark we reviewed relies on LLMs somewhere in the evaluation loop, either to generate questions or to score answers.

ExCyTIn uses models to turn incident graphs into Q&A pairs and to grade free-form responses, falling back to deterministic checks only in constrained cases. CyberSOCEval uses Llama models in its question-generation pipeline before shifting to algorithmic scoring. CTIBench relies on GPT-4-class models to produce CTI multiple-choice questions. CyberSecEval 3 uses LLM judges to rate phishing persuasiveness and other behaviors.

CyberSecEval 3 is a standout here. It calibrates its phishing judge against human raters and reports a strong correlation, which is a step in the right direction. But overall, we are treating these judges as if they were neutral ground truth. In many cases, the judge is supplied by the same vendor whose models are being evaluated, and the judging prompts and criteria are public. That makes the benchmarks simple to overfit: once you know how the judge “thinks,” it is trivial to tune a model or prompting strategy to please it.

That being said, “LLM as a judge” remains incredibly popular across the field. It is cheap, fast, and feels objective. It’s not the worst setup, but if we do not actively interrogate and diversify these judges, comparing them against humans, against each other, then over time we risk baking the biases and blind spots of a few dominant models into the evaluation layer itself. That is a poor foundation for any serious claims about security performance.

Technical Gaps

Even when the evaluation methodology is thoughtful, there are structural reasons today’s benchmarks diverge from real SOC environments.

Single-Tenant, Single-Vendor Worlds

ExCyTIn presents a well-designed Azure-style environment, but it is still a single fictional tenant with a curated set of attacks and detection rules. It tells us how models behave in a world with clean logging and eight known attack chains, but not in a hybrid AWS/Azure/on-prem estate where sensors are misconfigured and detection logic is uneven.

CyberSOCEval’s malware logs and CTI corpora are similarly narrow. They represent security artifacts cleanly without the messy mix of SIEM indices, ticketing systems, internal wikis, email threads, and chat logs that working defenders navigate daily. If the goal is to augment those people, current benchmarks barely capture their environment. If the goal is to replace them, the gap is even wider.

Static Text Instead of Living Tools and Data

CTIBench and CyberSOCEval are fundamentally static. PDFs are flattened into text, JSON logs are frozen into MCQ contexts, CVEs and CWEs are snapshots from public databases. That is reasonable for early-stage evaluation, but it omits the dynamics that matter most in real operations.

Analysts spend their time in a world of internal middleware consoles, vendor platforms, and collaboration tools. Threat actors shift infrastructure mid-campaign or opportunistically piggyback on others’ infrastructure. New intelligence arrives in the middle of triage, often from sources uncovered during the investigation. In that sense, a well-run tabletop or red–blue exercise is closer to reality than a static question bank. Benchmarks that do not encode time, change, and feedback will always understate the difficulty of the work.

Multimodality is Still Underdeveloped

CyberSOCEval does take an impressive run at multimodality, comparing text-only, image-only, and combined modes on CTI reports and malware artifacts. One uncomfortable takeaway is that text-only models often outperform image or text+image pipelines, and images matter primarily when they contain information not available in text at all. In practice, analysts rarely hinge a response on a single graph or screenshot.

At the same time, current “multimodal” models are still uneven at reasoning over screenshots, tables, and diagrams with the same fluency they show on clean prose. If we want to understand how much help an LLM will be at the console, we need benchmarks that isolate and stress those capabilities directly, rather than treating multimodality as a side note.

Modeling Limitations

Ironically, the very benchmarks that miss real-world workflows still reveal quite a bit about where today’s models fall short.

General Reasoning is Not Security Reasoning

CyberSOCEval’s abstract states outright that “reasoning” models with extended test-time thinking do not achieve their usual gains on malware and CTI tasks. ExCyTIn shows a similar pattern: models that shine on math and coding benchmarks stumble when asked to plan coherent sequences of SQL queries across dozens of tables and multi-stage attack graphs.

In other words, we mostly have capable general-purpose models that know a lot of security trivia. That is not the same as being able to reason like an analyst. On the plus side, the benchmarks are telling us what is needed next: security-specific fine-tuning and chain-of-thought traces, exposure to real log schemas and CTI artifacts during training, and objective functions that reward good investigative trajectories, not just correct final answers.

Poor Calibration on Scores and Severities

CTIBench’s CVSS task (CTI-VSP) is especially revealing in this regard. Models are asked to infer CVSS v3 base vectors from CVE descriptions, and performance is measured with mean absolute deviation from ground-truth scores. The results show systematic misjudgments of severity, not just random noise. This is an important finding from the benchmark

Those errors are concerning for any organization that plans to use model-generated scores to drive patch prioritization or risk reporting. More broadly, they highlight a recurring theme: models often sound confident while being poorly calibrated on risk. Benchmarks that only track accuracy or top-1 match rates will fail to identify the danger of confident, but incorrect recommendations, especially in environments where those recommendations can be gamed or exploited.

Conclusion

Today’s benchmarks present a clear step forward from generic NLP evaluations, but our findings reveal as much about what is missing as what is measured: LLMs struggle with multi-hop investigations even when given extended reasoning time, general LLM reasoning capabilities don’t transfer cleanly to security work, and evaluation methods that rely on vendor models to grade vendor models create obvious conflicts of interest.

More fundamentally, current benchmarks measure task performance in controlled settings, not the operational outcomes that matter to defenders: faster detection, reduced containment time, and better decisions under pressure. No current benchmarks can tell a security team whether deploying an LLM-driven SOC or CTI system will actually improve their posture or simply add another tool to manage.

In Part 2 of this series, we’ll examine what a better generation of benchmarks should look like, digging into the methodologies, environments, and metrics required to evaluate whether LLMs are ready for security operations, not just security exams.

This LABScon talk explores how hacktivist activity is strategically leveraged by nation-states and mercenary groups to obscure intent, destabilize targets, and weaponize public narratives. SentinelLABS’ Jim Walter draws on his decades of malware research and threat intelligence experience to decode the hacktivism ecosystem through a unique tooling-based analysis.

Using a four-tier framework for categorizing hacktivist groups, Jim describes a pyramid-shaped ecosystem that ranges from “commodity craptivism” at its bottom, characterized by high noise and low signal, to sophisticated state-front operations at the top, responsible for attacks with physical consequences timed to real-world events.

Jim explains why state-level threat actors increasingly adopt hacktivist personas. The motivations include plausible deniability, narrative control, and strategic influence operations designed to erode confidence in target regimes.

Through examples like Anon Sudan, Belarusian Cyber Partisans, NullBulge, and state-linked operations such as MeteorExpress and Handala, the talk reveals the distinguishing traits that separate top-tier actors from the rest. These indicators include consistent multi-year messaging, willingness to forego financial gain, sophisticated prepositioning capabilities, and measured communications crafted by professional writers.

The presentation concludes that most high-impact hacktivism reported today is actually “fictivism”, state-sponsored proxy operations masquerading as grassroots activism. With state actors leveraging this increasingly chaotic landscape to advance geopolitical objectives while maintaining deniability, this talk is essential viewing for anyone interested in the current hacktivist threat landscape.

About the Author

Jim Walter is a Senior Threat Researcher at SentinelLABS focusing on evolving trends, actors, and tactics within the thriving ecosystem of cybercrime and crimeware. He specializes in the discovery and analysis of emerging cybercrime “services” and evolving communication channels leveraged by mid-level criminal organizations. Jim joined SentinelOne following ~4 years at a security start-up, also focused on malware research and organized crime. Previously, he spent over 17 years at McAfee/Intel running their Threat Intelligence and Advanced Threat Research teams.

About LABScon

This presentation was featured live at LABScon 2025, an immersive 3-day conference bringing together the world’s top cybersecurity minds, hosted by SentinelOne’s research arm, SentinelLABS.

Keep up with all the latest on LABScon here.