Executive Summary

• SentinelLABS evaluated how frontier models reverse engineer binaries and annotate disassembler databases, finding recurring problems with inaccurate labels and incomplete analysis.
• To improve performance, we developed and tested a standard for LLM-driven reverse engineering that keeps annotations tied to evidence, verifies database changes, preserves project state, and requires review of the model’s work.
• We open-sourced VerifiedIDA, a reference implementation that applies the standard to autonomous reverse engineering in IDA Pro.
• Despite the growing capabilities of new models, our research shows that scaling reverse engineering requires that the analysis they produce remain inspectable, continuable, and reusable by humans, other models, and downstream security workflows.

Overview

The use of large language models (LLMs) for reverse engineering creates the possibility of examining binaries at a scale and speed exceeding that of manual analysis. It also makes reverse engineering accessible to those without specialized expertise in the discipline. The combination of scale, speed, and accessibility introduces a major shift across multiple domains. In cybersecurity, for example, defenders can analyze larger volumes of malware in less time, while attackers can misuse the same capability to considerably accelerate and scale vulnerability research and exploit development, shortening the time to exploitation and increasing the volume of intrusions.

As part of our broader innovation initiative for applying AI to automate and scale up defense workflows to keep up with an increasingly fast and widespread threat landscape, this work contributes to making LLM-driven reverse engineering inspectable, continuable, and reusable. These properties are important when reverse engineering output is used as part of cyber defense operations, such as incident response, detection development, or threat hunting, where analysts act on this output, share it with other stakeholders, and use it as input to other tools and workflows. This requires the ability to inspect the evidence behind an LLM's conclusions to verify them, have another analyst or model continue the work from where the LLM stopped, and reuse results coupled with the code and data they describe.

To this end, we focus on LLM-driven reverse engineering whose output is an annotated disassembler database, such as an IDA Pro, Ghidra, or Binary Ninja database. In contrast to a textual report, which reduces the analysis to a prose snapshot of the model's conclusions, detached from the code that supports them, an annotated database records findings directly on the disassembled code and data, as labels, types, data structures, and comments. 

As a result, an analyst can verify each conclusion against the code and data it describes. When the database is shared with other stakeholders, recipients get every annotation, not only the findings a report would contain, and can verify and use the parts they need. In addition, another analyst or LLM can continue the analysis from the existing annotations in the database or correct them if necessary, rather than interpreting a prose report and locating the code and data each conclusion refers to. Other tools and workflows can also work directly with the findings and the code they apply to, for example, to generate YARA rules that target functions identified as implementing specific capabilities, or to parse configuration data using recovered data structures. 

However, an annotated database provides these benefits only if its annotations are accurate and complete, which LLMs do not guarantee. To understand these limitations in practice, we conducted evaluation studies in which LLMs analyzed binaries and annotated IDA databases without safeguards for the accuracy or completeness of their work.

We first present the results of these studies, which show that models produced inaccurate labels, left some findings unrecorded, lost track of unfinished work, and stopped the analysis early. Full model specifics are provided in the section below.

Based on these findings, we propose a standard for LLM-driven reverse engineering that annotates disassembler databases. Under the standard, each annotation must be confirmed as recorded in the database and traceable to documented evidence against which its accuracy can be verified, outstanding work stays tracked until it is resolved, and an independent review assesses the accuracy of the annotations and identifies work left undone, returning its findings to the LLM that reverse engineers the binary for correction.

Finally, we present VerifiedIDA, a harness that serves as a preliminary implementation of the standard by mediating the LLM's access to an IDA database. VerifiedIDA is released as an open-source project, together with the specification of the standard and a user guide.

Evaluation Studies | LLM-Driven Reverse Engineering Without Safeguards 

The goal of our evaluation studies was to identify common challenges with model-led reverse engineering where the model had to annotate a disassembler database using basic tooling. A detailed comparison of how models perform, individually or by vendor or tier, was not our focus.

We used proprietary models from Anthropic and OpenAI in addition to open-source models to reverse engineer binaries of varying size, both malicious and legitimate. The majority of our testing presented here used GPT-5.6 Sol. The models worked through a simple harness, in which an LLM analyzes a binary autonomously, without any human or automated feedback and with access only to basic reverse engineering capabilities. The harness provides only general instructions and a fixed set of tools for viewing disassembled and decompiled code, following cross-references, searching the binary, and writing labels, types, and comments, such as free-text descriptions of functions, into an IDA database.

The full reasoning of Anthropic and OpenAI models is not available through their APIs, so we could not directly observe how they make decisions while reverse engineering a binary. Instead, we logged each analysis step by step in what we refer to as an LLM log, recording the tools the model called, the code it retrieved, its intermediate notes and planned next steps, and the edits it made to the database. An LLM log may show, for example, the model retrieving the pseudocode of a function, following its cross-references, and then labeling the function in the database. This allowed us to reconstruct what the model retrieved, what it based each annotation on, and how it acted on that information.

To illustrate some of the observations we discuss next, we present two case studies in the Case Studies section, each examining in detail how GPT-5.6 Sol performed a routine reverse engineering task. In both, the model reverse engineered a sample of ComRAT (SHA-1 hash: 0139818441431c72a1935e7f740a1cc458a63452), a backdoor attributed to the Russian APT group Turla. Case Study #1 covers the recovery of a data structure passed as a parameter to two ComRAT functions, whereas Case Study #2 covers the analysis of a single ComRAT function's behavior.

Structured Work Grounded in Evidence

We observed that the LLMs took a sound, methodical approach to reverse engineering tasks. For example, when labeling a variable or a function, they typically started with a broad analysis of the pseudocode or disassembly and then moved to targeted exploration, analyzing how the variable or function is used across the binary to determine the variable’s role or the function’s primary operation, which is what the label should describe.

The models also largely avoided fabrication. For example, when a model could not establish the role or type of a variable, it typically applied a generic label, such as unknown, or a generic data type, such as void, rather than inventing one the code did not support.

In addition, the LLMs made effective use of authoritative information relevant to the analysis, such as the public documentation of Windows API functions, which is highly likely present in their training data. For example, when a variable was passed to a Windows API function, they typed it based on the function's documented parameter types, and when a function called Windows API functions, they took the documented behavior of these API functions into account when determining the primary operation of the calling function.

Inaccurate and Incomplete Disassembler Databases

Alongside the strengths discussed above, we observed four major, recurring problems. These problems undermine the purpose of an annotated database as the primary output of LLM-driven reverse engineering, which is to serve as an accurate and complete binary-grounded foundation for any downstream use.

Accuracy of labels 

Some labels the LLMs applied to variables and functions did not accurately describe the variables' roles or the functions' primary operations. This matters because analysts often rely on labels to navigate an annotated disassembler database. Inaccurate labels therefore have consequences for anyone who uses an annotated disassembler database, human or model, whether to continue an analysis or to draw conclusions from it. For example, an ambiguous label may lead an analyst to redo work that is already done, while an imprecise one may draw attention away from binary behavior that matters. 

Inaccurate labels also misrepresent the binary under analysis, and any conclusion drawn from the database carries that misrepresentation forward. In addition, the inaccuracy of a label is not apparent from the database itself, since a label says nothing about how well it is supported. An inaccurate label is therefore either taken at face value, with the consequences that may bring, or caught only at the cost of redoing the analysis whose conclusion it was meant to record.

Capture of findings 

The models repeatedly established findings during the analysis without recording them as annotations in the databases, that is, they did not reliably translate reverse engineering findings into database annotations. These findings did not survive the analysis sessions, and the effort behind them was effectively lost, leaving the databases incomplete.

Continuity of work

As the analyses progressed, the models often lost track of unfinished work, which we suspect is primarily due to context compaction, the condensing of the models' earlier output and tool results to fit their context windows. Steps they had planned were dropped, leaving the parts of the binary they targeted unanalyzed and without annotations in the databases.

Completeness of analysis

The LLMs often stopped exploring before examining all the code relevant to what they were analyzing. For example, having settled on a function's primary operation, an LLM labeled the function without exploring all of the function's callees, some of which implement additional capabilities. These capabilities remained unidentified, and the function's label did not express them.

Reverse Engineering Standard

The four problems we observed are not variations of a single failure. Each points to a responsibility that cannot be left to the model alone.

Models are not infallible, so no system can guarantee that every label they apply is accurate. A system can, however, keep the evidence behind each annotation, so an inaccurate label can be found and corrected without redoing the analysis it was meant to record. It can confirm which findings reach the database, carry unfinished work beyond the model's context, and check whether the code behind a conclusion was examined.

These checks fall into two categories. Mechanical checks can be made without interpreting the code: whether a requested change persisted, or whether a function's callees were inspected before the function was labeled, as in Case Study #2. Semantic checks require review on whether a label describes a function's primary operation, or whether the analysis establishes how two capabilities relate.

This is the division of labor at the center of our standard: the model makes the analytical judgments, and the system around it preserves the evidence, maintains the state of the project, and enforces follow-through. We formalize this into four requirements.

Problem	Requirement	What the system must establish
Accuracy of labels	Attributable evidence	What each annotation is based on, so its accuracy can be verified
Capture of findings	Verified changes	What was requested, what was recorded, and whether it persisted
Continuity of work	Preserved project state	What is known, what has been done, and what is planned next
Completeness of analysis	Enforceable scope and review	What must be examined, and what remains unresolved

Attributable evidence. Every annotation must be traceable to the evidence the model retrieved to support it, identified by the binary and the state of the database it came from. Evidence that no longer reflects the current database must not support new annotations.

Verified changes. The system must check what each requested change recorded in the database and whether it persisted. If a change fails, the accepted database must stay intact, and the model must receive feedback it can act on. This matters most for type changes. A recovered structure that cannot be applied cleanly, or that alters parts of the database the model did not intend to change, can be detected and rejected mechanically before it affects how other code decompiles.

Preserved project state. Accepted annotations, the history of changes, the working summary of the binary, and the list of pending tasks must be stored outside the model's context. The analysis can then be resumed after context compaction, an interruption, or a switch to another binary, by the same model, a different one, or a human analyst.

Enforceable scope and review. An independent reviewer must be able to challenge the accuracy of annotations and identify work left undone. Its findings must go back to the investigating model and stay tracked until each is corrected, rejected with evidence, or recorded as unresolved for a human to pick up.

Our proposed standard is the floor for LLM-driven reverse engineering. It does not make a model's analysis correct, and it does not prescribe how a model should reverse engineer a binary. Many tools already use models for reverse engineering in their own ways, and more will follow. Verification is essential, but it establishes only what was recorded, not whether it is right, and review by another model or by an expert human can still miss errors. What the standard ensures is that the model's conclusions remain tied to evidence that can be revisited, and that its mistakes are identifiable rather than hidden. The standard is also independent of a particular disassembler. We publish it as an interface specification alongside our reference implementation for IDA Pro, which we describe next.

The VerifiedIDA Harness

VerifiedIDA is our reference implementation of the standard. We built it for IDA Pro first because it is the disassembler our team works with most. The model chooses what to investigate, inspects code, and proposes annotations. VerifiedIDA acts as the host and mediates every access to the IDA database: it identifies the target of each request, requires current evidence, applies edits to a candidate copy of the database, checks the result, and records each operation. It maintains three records:

• IDA databases, which hold the accepted labels, comments, types, interfaces, and relationships, one per binary, including embedded binaries.
• A SQLite ledger, which holds inspections, operations, verification results, database revisions, and tracked work.
• A project notebook, which holds the model's current understanding, uncertainties, and next actions, along with a journal of how they changed.

Attributable evidence. Every edit must cite evidence the host recorded for the current target and database revision. Evidence is identified by binary and revision rather than by address alone, because a binary and the binaries embedded in it can use the same addresses. If the database has changed since the model inspected the code, the host rejects the edit and explains how to obtain current evidence. Model inferences and historical evidence are labeled as such. A reviewer can start from any annotation, find the operation that introduced it in the ledger, and examine the code the model was shown when it made it.

Verified changes. The host applies each edit to an isolated candidate database, reads the result back through a separate path, and checks it before the candidate becomes the accepted database. Type and prototype edits also receive a comparison of the affected IDA state, and unrelated changes lead to rejection. If an edit fails, the accepted database is left unchanged, and the model receives the target, the failed stage, the requested and observed state, and a recovery direction. Checkpoints reopen the database in a fresh IDA process to confirm that accepted changes persisted.

Feedback also targets the capture problem directly. After an edit, the host flags incomplete annotations, such as a renamed function without a behavior comment or parameters that retain generated names, and reports where a newly created type is not yet applied. In our worked example, based on the ComRAT structure from Case Study #1, the feedback reported that the structure the model had created was declared but not used anywhere, and the model went on to apply it to the function's parameter.

Preserved project state. The notebook records where the investigation stands, and its current-state sections are rewritten as the model's understanding changes. After context compaction, the model is given its current instructions and the recorded project state, and any further edit still requires fresh evidence from IDA. When the model follows an embedded binary, the host records the child's bytes, how it was extracted, and its link to the parent, and opens a separate database for it. Before switching, we require the model to record what it has established in the parent and how the parent uses the child, so it knows where to return.

Enforceable scope and review. Independent review starts with the host building an inventory of annotations and possible gaps. Reviewer models then examine individual claims and the overall explanation of the program, with read-only access to disposable copies of the databases. The host validates their findings and freezes the scope of required corrections. The investigating model then resumes and works through one finding at a time: it can apply the correction, revise it, reject it with evidence, defer it, or request a specific follow-up. It then checks that the accepted review conclusions agree with the annotations, and any remaining disagreement blocks completion. The final report states what was verified, what remains unresolved, and whether the completion policy was met.

Coverage mode. We reran the Case Study #2 function with an experimental call-flow mode enabled. In six earlier GPT-5.6 Sol runs, none had opened sub_180081450, and the four that annotated the parent all omitted forwarded-export handling. 

In the new run, Sol began from a clean database and was not told which callee mattered. Once it committed a claim about sub_180077C10, the mode listed the function's 20 direct internal callees and held that claim open until each had a recorded outcome. Sol opened sub_180081450, identified it as the recursive resolver for forwarded exports, and confirmed a parent comment that separates local export-table parsing from the recursion it delegates. 

A repeat under a revised policy reached the same result across all 20 callees, and every edit survived a fresh-process reload. The mode follows direct calls only. An early attempt to apply it across the whole project generated 265 required nodes for eight edits, so we reserve it for selected claims.

Conclusion

Over the past year we’ve watched the infosec community grapple with the rapid adoption of AI tools and increased automation. It is clear that AI models can now reverse engineer binaries at a scale no team of analysts can match. As the volume of analysis the models produce grows, it creates a widening gap in our ability to understand how decisions were reached and what evidence they were based on. After all, a model's labels, types, and descriptions are only as useful as an analyst's ability to confirm them, correct them, and build on them. 

A database makes this level of inspection possible in a way that a report cannot because each conclusion can be tied to the code it describes. This allows a person (or another model) to easily continue the work or incorporate the findings into downstream detection and hunting workflows. 

LLMs give us the opportunity to automate more security analysis but only insofar as the work they produce remains understandable. Scaling the analysis without also scaling our ability to inspect it replaces one bottleneck with another.

Our evaluation studies showed that models alone do not reliably produce attestable databases. Labels drift from what the code does, findings stay in logs, planned work disappears, and analysis stops before the relevant code has been read. The standard and VerifiedIDA respond by leaving the analytical judgment with the model and giving the system around it the job of keeping that judgment checkable. Our rerun of Case Study #2 shows the difference: the helper that six earlier runs skipped became work the model could not leave unfinished.

We are releasing both as a starting point. The requirements describe the minimum a model-driven workflow should provide, and our implementation covers a single disassembler. We invite the community to test the specification against their own tooling, build adapters for Ghidra and Binary Ninja, and push back on requirements that do not hold up. Where sharing is permitted, we also encourage researchers to publish annotated projects alongside their reports, so others can extend an analysis instead of repeating it.

This paper complements research we presented at LABScon 2026.

Case Studies

#1 | Recovering a Data Structure

Recovering data structures is routine reverse engineering work. At the memory level, a structure is a base address with fields at fixed offsets from it, where a field may be a complex data type itself. If the binary carries no debug symbols, it does not declare field offsets, types, and names, and these have to be recovered through analysis.

Two ComRAT functions, with the IDA default names sub_18002C770 and sub_18002CDA0, receive a pointer to a data structure as their parameter. IDA types these parameters as an integer.

The LLM typed the parameters as a pointer to a data structure, which it declared as follows.

The LLM approached the recovery of the structure in two stages. First, from the pseudocode of sub_18002C770, sub_18002CDA0, and the functions they invoke, it observed references to specific offsets from the function parameter value (the structure base address), concluding that the parameter is a pointer to a structure. It then widened the analysis to the rest of ComRAT's code, looking for references to the structure's fields wherever they occur. For each field it identified, it analyzed the context in which the field is used to propose a label, type, and size. 

This enabled the LLM to recover the full set of structure fields, and from their sizes the correct total size of the structure itself.

The strongest evidence the LLM had for typing a field was its use as a parameter to a Windows API function. Many Windows API functions are documented publicly and their parameter sets are relatively stable, so the types of these parameters are highly likely present in the model's training data. IDA may also expose these types in the pseudocode provided to the LLM since it auto-defines parameter types for many of these functions. The model could therefore determine field types from authoritative sources.

For example, the LLM observed that a pointer to the structure field at offset 0x30 is passed as the tenth parameter of CreateProcessW. Microsoft documents the type of this parameter as a pointer to a PROCESS_INFORMATION structure, and IDA's pseudocode shows the same type. The LLM used this information to correctly type the field as PROCESS_INFORMATION.

The LLM did not produce annotations that its analysis did not support. For example, it found no use of the field at offset 0x28 within the functions under analysis and did not label the field after an invented role. Instead, it flagged the finding and decided to explore ComRAT's code further to establish the field's role. It found no use of the field anywhere in ComRAT and labeled it unresolved_28. Our own analysis confirms that the field is indeed never used.

However, several structure field and function labels produced by the LLM are inaccurate, since they do not describe what they should: the role a field plays in the code or the primary operation a function performs. For example, the LLM labeled the field at offset 0x28 unresolved_28, naming its own analytical state rather than the role of the field, or the lack thereof. The label is also ambiguous, because it does not distinguish a field that has not been examined from one that has been examined and found to be unused. In the case of this field, a name such as unused would be more appropriate.

The LLM also named the function sub_18002C770 PrepareSuspendedSurrogateProcess, although the function’s primary operation is the creation of a browser process in a suspended state. The LLM’s label does not indicate process creation at all, and Prepare suggests work that precedes a later step rather than the function’s primary operation. The LLM’s free-text description of the function (“Create the configured surrogate process before proxy injection”) shows that the model had identified the primary operation. The description leads with it and treats the injection that follows as context, whereas the label PrepareSuspendedSurrogateProcess keeps only the context and leaves out the process creation. A human reverse engineer would be more likely to apply a label such as CreateSuspendedBrowserProcess, which is more descriptive and more accurate in the context of the task.

Beyond inaccurate labels, the LLM did not record all of its findings in the IDA database, as the annotation of the structure field at offset 0x0 demonstrates. This field is a pointer to a structure, which holds data associated with a log writing mechanism, and the LLM logs show that the model identified the field as such.

However, in the IDA database, the LLM typed this field as a generic void pointer and labeled it owner_context rather than something closer to its role, such as LogWriter. The LLM had established the type and role of the field but did not translate these findings into IDA database annotations.

#2 | Analyzing a Function’s Behavior

A ComRAT function, with the IDA default name sub_180077C10, retrieves by name the address of a function exported by a dynamic link library (DLL) loaded in a remote process. If the export is forwarded to another DLL, the function resolves the address of the forwarded export recursively.

The LLM labeled this function ResolveRemoteExportAddress, correctly expressing its primary operation. As in Case Study #1, the strongest evidence the LLM relied on was the Windows API calls the function invokes. ResolveRemoteExportAddress also compares values against publicly documented constants, which provided the LLM with further authoritative evidence of the function’s operation.

For example, from the ReadProcessMemory calls, the LLM established that the function reads memory in the address space of a remote process. From the comparisons against the MZ (0x5A4D) and PE (0x4550) signature constants, the model determined that the function looks for Windows PE (Portable Executable) headers. Because the PE format is publicly documented and highly likely present in the model’s training data, the offsets into these headers visible in the pseudocode enabled the model to identify which of the DLL’s PE structures the function reads, such as its export directory and the array of names of its exported functions.

The LLM's free-text description, “Resolve an exported function address in a remote module from its PE export directory”, is accurate but incomplete, since it does not account for the resolution of forwarded exports.

If an export is forwarded to another DLL, sub_180077C10 (ResolveRemoteExportAddress) calls sub_180081450, which resolves the address of the forwarded export. Treating its understanding of sub_180077C10 as complete, the LLM labeled and described the function without exploring all of its callees, including sub_180081450. The capability implemented by sub_180081450 therefore remained unidentified.

Hackers captured a Flock camera and got a look (alternate link) at the software:

While much of the automatic license plate reader’s (ALPR) most sensitive storage remained encrypted and inaccessible, the joint analysis of the recovered data shows that software running on the device explicitly detects people as well as vehicles, license plates, and bicycles. The camera can produce dozens of images of a single passing vehicle and, according to several weeks of recovered logs, generated more than a million images. Its computer-vision software also sometimes isolated bumper stickers and other graphics, including, in one case, an American flag patch on a motorcyclist’s saddlebag.

If you’re wondering how the hackers got by disk encryption, one of the unencrypted partitions contained the key for an encrypted partition. That’s pretty bad security engineering.

Executive Summary

• SentinelLABS developed a multi-stage reverse-engineering benchmark for the latest generation of frontier models by recreating our recent investigation of fast16, a unique 2005 sabotage implant.
• Most AI benchmarks test bounded tasks. This benchmark tests whether a model can keep a malware investigation trustworthy as new evidence repeatedly invalidates its earlier conclusions.
• OpenAI’s GPT-5.6 Sol was the only publicly available model to complete the full eight-stage investigation, giving concrete shape to what ‘Frontier-class’ capabilities offer analysts. GPT-5.5, GLM-5.2, and the Opus 4.x family produced capable local analysis but could not carry it through the gradient.
• What distinguished the completed runs was project-scale recovery: withdrawing contradicted conclusions, repairing technical artifacts, and updating dependent reporting without losing the investigation.
• Senior reverse engineers remain essential. Even the strongest runs made semantic errors, accepted weak quality controls, and claimed readiness prematurely. We assess the best current use as supervised investigative agency, with human analysts defining objectives, exposing blind spots, and retaining final publication authority.

Beyond Vulnerability Discovery

Since ChatGPT arrived in late 2022, we have been bullish on what large language models could do for reverse engineering and malware analysis. The early models were useful for teaching but too rudimentary for production work; that changed with the advent of reasoning models. OpenAI’s o1-preview, in September 2024, was the first to show the kind of sustained problem-solving the work demands, and within months Sean Heelan had used o3 to find a net-new vulnerability in the Linux kernel. In cybersecurity, though, our understanding of what these models can do remains stovepiped to vulnerability discovery.

The frontier labs took on vulnerability discovery deliberately, because that competency keeps agentic code generation from quietly shipping vulnerable code at scale. OpenAI built Aardvark, since folded into Codex; Google DeepMind announced Big Sleep, available internally to its Project Zero researchers; and Anthropic followed with selective access to Mythos Preview. Concerns that these capabilities could be misused have led the labs to stricter guardrails and ‘know your customer’ style controls that limit access to specific capabilities, or to entire model variants.

OpenAI’s Daybreak initiative and its Trusted Access Program opened a dedicated variant, GPT-5.*-cyber-preview, with guardrails relaxed for cybersecurity use cases, while Anthropic’s Glasswing initiative and its Cyber Verification Program provided early access to Mythos Preview and the promise of lesser guardrails respectively. For a short period in mid-June 2026 access to the highest-end flagship models from both providers required some form of U.S. government clearance. At the time of writing, GPT 5.6 Sol is widely available, while Mythos 5 still requires clearance and access as a Glasswing partner.

The existence of this new class of models left us with an unusual task: benchmarking what these models can actually do on the work defenders care about, and assessing whether they live up to the surrounding hype. If they do, we have to reckon with what that means for malware analysis and reverse engineering, disciplines that until now have been limited mostly by how little expertise exists relative to the collective need.

A Benchmark Built From a Real Investigation

We recently published our research on fast16, a 2005 Windows toolkit built to sabotage high-precision solvers used to model nuclear-weapons behavior. The sample provided an ideal test case because its layered design punishes shallow analysis.

On the surface, svcmgmt.exe appears to be a Windows service implant with two embedded components. Their combined logic becomes visible only through a higher layer of abstraction, an encrypted Lua-driven operations framework. A kernel driver easily mistaken for a filesystem rootkit actually contains a patching engine with 101 rules, and those rules must be repurposed to identify candidate target software before an analyst can extrapolate the approximate effects of the sabotage operation. This investigation is a daunting task for human experts, and reproducing it autonomously should be considered a significant milestone for artificial intelligence.

When Vitaly Kamluk shared our discovery of fast16 at Black Hat Asia, he pointed to the failed attempts of Opus 4.6, GPT-5.4, Grok 4.2 Reasoning, Gemini 3.1 Pro, and DeepSeek Reasoner to triage the sample autonomously. At best, the models concluded fast16 was a rootkit and could not move past that assumption, converging on the same plausible but inaccurate conclusion most human reverse engineers reached on first contact.

Used instead as assistants, the models earned their keep. Opus 4.6 and GPT-5.4 compressed what was originally a three-week dedicated expert effort by Kamluk into an 8-10 hour task manageable by a less experienced reverse engineer. Autonomous investigation was out of reach when we began this work; co-intelligence was the ceiling.

Standardizing the Benchmark

We codified our reverse-engineering methodology, inspired by Rolf Rolles’ approach to statically analyzing binaries, into a skill the harness loads on /reverse, which removes prompting variability across models. Each run gives the model a sample, IDA Pro 9.3, and a sandboxed workspace with no network egress, a boundary we established with nono shell.

The model must produce fully annotated IDA databases by renaming variables, recasting data types, categorizing functions, recovering hidden components, and folding those findings back into the host-sample database.

Only after producing a gold-master IDB for approval does a run advance to walking the execution flow and detailing findings, with every claim anchored to specific addresses and artifacts so the work can be independently verified. Judging the model on a refined IDB rather than a prose report closes off the deductive shortcuts and approximations typical of LLM malware triage.

We took an ecological approach, a bounded case study that benchmarks a capability profile in a complex real-world scenario rather than feeding another general leaderboard or isolating a single narrow competency. We also deliberately tested the combination of model and harness, since agentic workflows are how practitioners actually use these systems, and most benchmarks still test models in abstraction.

Runs level up through eight stages, each introducing new artifacts and minimal prompting for the new task:

Measuring ‘Intelligence’

When Nick Fountain interviewed us for NPR’s Planet Money, he observed that fast16’s brand of sabotage amounts to epistemic warfare, an attack on the target’s confidence in its own ability to know something to be true. There is a fitting symmetry in using the same sample to examine what the frontier labs sell as intelligence, because the difficulty there is also epistemic. ‘Intelligence’ as a concept bundles many interwoven capacities, and model tiers simplistically priced as if intelligence gets cranked up with a simple knob tells us nothing about what any one offering is worth on real work.

Many of those capacities already have familiar names, and the names do finer work than a score. Take thoroughness, the completeness of a particular piece of work. An analyst who consistently applies that effort where it matters is diligent. One thorough artifact does not make a diligent analyst; diligence is the longitudinal quality, and thoroughness one of its observable expressions.

Exhaustive coverage is not diligence either. An analyst who inspects everything indiscriminately while neglecting the consequential work is being pedantic, and their effort is wasted on the broader investigation. Diligence, in short, is sustained, priority-sensitive thoroughness.

Many similar qualities sit inside what we call intelligence, and a meaningful benchmark must account for them as we would with a human analyst, by characterizing behavior across an entire investigation rather than compressing it into a rank.

We scored two different things. Local analytical quality asks whether the model made technically sensible decisions with the evidence available. Investigative horizon asks how long it preserved that quality as the scope, evidence, and reporting obligations expanded. The two are related but not the same, and the gap between them is where most models failed.

What the Runs Showed

GPT-5.5 produced long autonomous runs, sometimes grinding uninterrupted for a day or two, with results relatively consistent across reasoning-effort settings. It did meaningful, semantically conservative work refining IDBs and repeatedly rejected edits that improved a count while making the decompilation less truthful. Its weakness was aggregation. Residual cast debt, function-tree defects, or incomplete cross-component verification kept it from clearing broad evaluation gates, and it never converged its extensive work into a complete package for review. Notably, nominal effort settings and multi-agent use showed no monotonic relationship to clearance. GPT-5.5 never escaped the grind phase of level 1.

We ran a single GLM-5.2 run through a serverless deployment on Fireworks AI with a $100 budget. The run showed measured local experimentation and capable backup-based recovery, which it needed after the most severe promotion failure we observed, losing 113 embedded functions and having to restore them before working again. One run is not a characterization of the model, but it is enough to temper claims of GLM-5.2 performing at the level of the frontier completers.

Opus 4.7 and 4.8 advanced further, showing capable technical correction and evidence-responsive revision, including a strong example of converting a speculative relationship into a structural test and rejecting it. Their recurring limitation was milestone overpromotion, a familiar eagerness to convert favorable local results into readiness claims before visible defects were resolved. They required operator challenge at important quality boundaries and did not sustain closure through the full benchmark. We are not altogether certain these newer iterations are in fact performing better than the earlier Opus 4.6 for our use cases.

The common pattern across this tranche was local competence without dependable project-wide closure. These models performed many of the individual behaviors seen in the strongest runs. What they did not demonstrate was the same behavior recurring through the full eight-stage investigation.

GPT-5.6 Sol | A New Class of Contender

When GPT-5.6 Sol was introduced, we seized the opportunity to evaluate it against our real-world benchmark. As the name implies, we expected incremental improvement from GPT-5.5 but instead found the model operating in a league of its own, with all three GPT-5.6 Sol runs completing the benchmark at different effort levels.

Sol was far from infallible. Its runs declared checkpoints clean or complete before later review found meaningful problems. The distinctive pattern was what happened next. Sol repeatedly responded to a local failure with a structural fix, reassessing the quality-control process that had let the mistake through and building scaffolding, such as codifying file hashes for even routine operations, to keep entire classes of failure from recurring.

The xhigh run provides the clearest example because its error began with a valid observation. fast16 contains a program of 101 byte-pattern rules that recognize locations inside an executable, capture addresses, replace code, and repair the executable metadata those changes require. Replaying that rule program across a larger software collection, Sol found that some rules fired not only in LS-DYNA, the engineering package fast16 targets, but also in MOHID and PKPM, two applications built from a similar Fortran lineage. It proposed that fast16 might be a general compatibility component rather than a tool engineered for sabotage.

The collisions were real, and we made similar observations early in our own investigation, but the explanation gave too much weight to the mere presence of matches. The model corrected course by demanding greater rigor of itself, asking whether the matches produced a meaningful modification sequence. In LS-DYNA they did. In MOHID and PKPM they produced only haphazard fragments, often triggered by generic compiler-generated code.

What first looked like broad compatibility was better explained as collateral risk from loose rules. Sol not only self-corrected but found better footing for that correction than we did. It then withdrew the compatibility hypothesis, preserved the valid measurements, restored the LS-DYNA modification mechanism as the authoritative account, and marked the rejected branch as historical provenance so later analysis would not inherit it.

5.6 Sol’s performance has pushed the boundary of the possible for automation in cyber threat investigations. Given its ready availability, we are interested in seeing how these capabilities enable the practitioner ecosystem to advance the state of the art in the same manner as alternate research disciplines like math and physics.

Project-Scale Recovery

What separated the completed runs from the rest was not technical insight, error detection, or willingness to revise. Every cohort supplied examples of each. The recurring difference was the scale at which recovery occurred. Across a long-horizon investigation, the completed runs did more than correct the sentence or script in front of them. They rebuilt the part of the project that later work would trust, in a repeatable pattern:

1. Withdraw the contradicted claim rather than softening its wording while continuing to rely on it.
2. Map the blast radius, identifying every conclusion, artifact, and test that depends on the disputed result.
3. Repair the cause, correcting the analysis or the quality-control gap that let the error through.
4. Carry the correction through the files another analyst or later stage will actually use.
5. Reopen the artifacts and run a check capable of disproving the corrected claim.
6. Separate work that blocks the current conclusion from uncertainty that can be disclosed and deferred.

That last step is easy to underestimate. An investigation can fail by stopping too early, and it can also fail by expanding forever. Deciding which unresolved questions block a conclusion is what convergence means in this benchmark.

This is not a single new reasoning faculty. It is the coordination of familiar capabilities over time, spanning correction, scope control, verification, state management, synthesis, and stopping judgment. The evidence supports project-scale recovery as a common feature of the completed runs. It does not prove a permanent intelligence threshold, and looking only at finishers cannot explain why every other trajectory fell short. The number of runs per family is small, and operator involvement was consequential throughout. What survives those caveats is the modest formulation that the completed runs repeatedly re-established a trustworthy investigative state after later evidence destabilized it.

Notes on Cost

Across all experiments, our runs consumed more than 23 billion tokens across model families and effort levels. The aggregate cache read rate was 93.58%, translating to different discounting rates across providers. Our self-hosted GLM-5.2 run offers a floor for comparison, producing 267.41M tokens on a $100 budget and a decent level-1 attempt charitably comparable to some GPT-5.5 and Opus 4.7 performance.

What These Results Mean for Analysts

The practical lesson stops well short of replacement for senior reverse engineers. The strongest runs still made serious semantic errors, accepted weak quality controls, promoted work too early, and depended on operator review at consequential moments. The new capability is that some runs remained useful after those failures. They owned substantial investigative workstreams, recovered from invalidated theories, preserved the valid parts of earlier work, and continued into publication-oriented analysis without starting over.

That changes the role these systems can plausibly play. A weaker model may be valuable as a local copilot, explaining a function, testing a hypothesis, or drafting a section. The strongest observed runs could act more like supervised investigative agents, maintaining and repairing a larger body of work across days, artifacts, and changing evidence. The supervision qualifier remains essential. Humans defined the investigative objective, controlled evidence release, exposed important blind spots, decided what quality meant, and retained final publication authority.

The scale of the shift deserves emphasis. Reverse engineering and malware analysis have been kept from scaling for three decades by the scarcity of expert practitioners. That constraint does not disappear, since expert guidance still determines whether the work can be trusted, but a single expert supervising these systems can now expand their output by any metric we track.

Conclusion

The line the benchmark exposed ran between local competence and sustained ownership of an investigation, rather than between smarter and lesser models. GPT-5.5, GLM-5.2, and the Opus 4.x family frequently produced valuable analysis. GPT-5.6 Sol also made meaningful errors. What distinguished the completed runs was their repeated ability to recover and diffuse that correction throughout the whole project, rather than just confronting their immediate context.

Having completed this stage of our evaluations, we took the best candidate produced by 5.6 Sol High and put it through an extensive adversarial refinement process using an ensemble of models to produce the highest automatable quality of IDBs and investigation artifacts. We are sharing these so other researchers can verify the work output against the sample and build on them rather than start from the raw binary. We make no claim that it is perfect, only that it is the greatest level of automated refinement the most capable models achieved through proper methodology, repeated standard enforcement, and a lot of tokens burned.

That is a less dramatic claim than a new form of autonomous intelligence. It is also more useful. For defenders, the question that matters most is whether the models can reliably preserve and advance a trustworthy investigation. Once we can count on that, it’s up to us to redefine a greater scope of impact for a new frontier of cyber threat investigations.

Resources

fast16-idbs – Gold-Master IDA Databases