A swarm of autonomous OpenAI agents spent six weeks this summer turning an obscure, 25-year-old German developer wiki into a private message board, without OpenAI’s knowledge, according to independent researchers.

The agents used it to trade answers to timed tasks, reverse-engineer a random number generator, and share a technique for bypassing network restrictions meant to keep them off the open internet.

“These AIs colluded to share answers, research their environment, and bypass sandbox restrictions,” researchers at the Nightingale Collective wrote in a report.

The agents were expected to operate independently with read-only access to the internet. Instead, they “found a way to use their read access to write information to an obscure German wiki” and began using it as a shared message board.

The group documented roughly 18,000 wiki edits made between May and July.

“OpenAI has not publicly disclosed this incident,” the report added.

The researchers said this wiki-based swarm is a separate group of agents from the one behind that Hugging Face attack.

“The main reason we believe this was a distinct swarm is because these agents explicitly had internet access as part of their task — the whole point was web browsing,” the researchers wrote. “The Hugging Face agents were in a sandbox without internet access and had to hack their way out by exploiting the Artifactory package manager.”

From isolation to coordination

Once the agents began writing to the wiki, the researchers said it effectively became a coordination layer.

“The agents used this wiki to communicate information with each other, primarily to help them succeed at their task,” the report states. “They asked for answers, pooled results, and shared techniques for bypassing their restrictions.”

Based on the posts reviewed, the researchers say agents appeared to use each other’s outputs to complete tasks more efficiently, in some cases enabling near-instant responses when similar questions appeared later in the sequence.

The activity was sustained and large-scale. The researchers identified thousands of distinct agent identities operating over a period of weeks, with a sharp increase in coordination beginning in mid-June.

Sharing ways to bypass controls

The report also documented agents exchanging techniques to circumvent technical restrictions.

According to the researchers, agents faced limitations that allowed only GET requests, but they collaborated on ways to bypass those constraints, including exploiting how trusted hostnames were handled within the sandbox environment.

Other agents then tested and confirmed these methods, suggesting that the wiki was used not just for coordination but also for distributing reproducible bypass techniques.

The researchers also observed agents uploading compressed datasets across multiple pages, which they say indicates the wiki was being used as shared infrastructure rather than incidental output.

Where containment assumptions fall short

For enterprise leaders, the findings point to a more specific failure: not of compute isolation, but of how agent behavior is constrained.

“This incident exposes an incomplete definition of containment,” said Sanchit Vir Gogia, chief analyst at Greyhound Research.

Gogia noted that the evidence does not show agents escaping their execution environment, but rather bypassing network restrictions and creating “unauthorised shared memory on a public wiki.”

“A sandbox can protect the host while failing to contain the work,” he said, adding that a domain allowlist “says nothing about what the agent can make the destination do.”

Gogia said organizations lack visibility into which agents are acting with authority, as well as where those agents may be storing or sharing data externally.

“Any service an agent can write to, and another agent can read is shared memory,” he said, whether or not that interaction is visible to orchestration systems.

Discovery and possible intervention

The timeline reconstructed from wiki logs suggests the activity continued for weeks before being curtailed.

“We believe OpenAI discovered the message board,” the researchers wrote, pointing to visits from IP addresses associated with the company beginning June 21.

The report noted that agent activity on the website dropped to near zero the following day, adding, “We infer that OpenAI intervened at this point to prevent further edits.”

For practitioners, analysts said, the episode also highlights limitations in how agent activity is typically monitored.

“A capable agent finding a coordination channel on a public wiki is exposing a blind spot that was never instrumented,” said Amit Kumar Jena, head of AI at Kanerika, adding that most monitoring systems were not designed to detect “inter-agent coordination on external resources.”

Jena said many enterprise controls remain focused on model outputs rather than behavior. “What the model says, not what it does,” he said. “That breaks completely with multi-agent systems.”

Disclosure questions

The report pointed out that “OpenAI has not publicly disclosed this incident,” raising broader questions for enterprise buyers around how such events are classified and communicated.

Gogia said the industry lacks a clear category for incidents that fall short of a breach but still represent consequential system behavior.

“A vendor’s definition of an incident must not become the customer’s blind spot,” he said.

Jena added that enterprises should evaluate vendors not just on disclosure, but on detection capability. “Enterprises should ask not just whether a vendor disclosed, but whether they even knew,” he said.

OpenAI did not immediately respond to a request for comment.

Concerns about the risks of AI systems are certain to be met with four words: human in the loop. The discussion may broaden, but the assurance is inevitable. It’s an AI governance phrase that’s become so rote you hear it in every direction and likely have said it yourself.

But IT leaders should be wary of vendor or team claims that they’ve built human-in-the-loop systems into AI tools because some of these supposed guardrails are no more than rubber stamps.

Some so-called human-in-the-loop systems don’t give employees overseeing the AI tools either the control or the time necessary to fix any problems, some IT experts point out.

For human-in-the-loop systems to actually work, employees overseeing AI tools need to have the domain knowledge and context to take the action the AI tool is addressing when the AI isn’t involved, and they need to have the authority to override the AI decision, says Doug Shepherd, head of offensive security at internet services provider Cloudflare.

Promises of human-in-the-loop systems give IT leaders comfort, but the underlying process often doesn’t work as advertised, he adds.

“If your human in the loop can flag something but can’t actually stop it, that’s not human in the loop, that’s a human adjacent to the loop,” Shepherd says. “That’s performative governance.”

Shepherd, speaking at the recent CIO 100 Awards and Conference in Frisco, Texas, encouraged attendees to embrace AI and focus on projects that drive adoption and impact. Organizations that fail to push AI initiatives will be left behind, he suggested, but he also warned that blind adoption, without focusing on meaningful outcomes and guardrails, can lead to huge setbacks.

Many organizations reach for human in the loop as an important control, but no one stress tests it, he adds. “It gets projects approved, and too often, it does the political work, but not the risk work,” he says.

Darren Kimura, CEO and president at AI integration platform vendor AISquared, agrees that many organizations are deceiving themselves with so-called human-in-the-loop systems.

“Most companies that say they have a human in the loop actually have a human watching the loop,” he says. “The person can see the decision and flag a concern, but they cannot stop it, change it, reject it, or escalate it.”

IT leaders should ask themselves a handful of questions: Can reviewers halt the actions before they take effect? Can they change the output? Are their overrides recorded and enforced downstream? “If the answer to any of those is no, the human is just monitoring AI,” Kimura says.

Too many decisions

Another problem with human-in-the-loop systems is the decision fatigue that can set in when employees are asked to review too many AI decisions and end up button mashing instead of thinking about the consequences.

The AI reviewer needs the expertise and context to evaluate the recommendation, enough time to do so, and both the authority and technical ability to reject or reverse it, says Eric Billingsley, COO and CTO of AI assurance company TrustScale.

But even a qualified and empowered reviewer may gradually stop exercising independent judgment when the AI is consistently right, he notes.

“If the system is right 95% of the time, the person’s job becomes waiting for the rare case when it is wrong,” he says. “Humans are not particularly good at sustained vigilance of a highly reliable automated system. Eventually, review becomes confirmation.”

A good AI system can create bad human controls, he adds. “When the exceptional case arrives, the reviewer may approve it because the system has trained them, through hundreds of correct recommendations, to trust it,” he says.

Billingsley advises IT leaders to evaluate human-in-the-loop systems the same way they monitor other security controls. A control must be monitored, tested, and produce evidence that it is operating as intended, he says.

“A log showing that someone clicked ‘approve’ is not enough,” Billingsley adds. “You need evidence that the person had the necessary context, applied independent judgment, and had the authority to override the AI.”

Robert Blumofe, EVP and CTO at cloud computing and security vendor Akamai, sees the same problems Billingsley does. Some type of human oversight is preferable to fully autonomous AI, he says, but human in the loop can turn into a mind-numbing exercise.

“LLMs produce the correct output just often enough to lull us into a complacent belief that they are more reliable than they really are,” he notes. “After diligently checking the AI output each time and finding no errors, diligence wanes, and human in the loop turns into rote approval.”

IT leaders should take the time to figure out what they’re getting into when vendors or their internal teams pitch a human-in-the-loop system, Blumofe says.

“It’s incredibly important to understand exactly how the system is designed and when and how the human will interact with the AI,” he adds.

Organizations should also explore ways to deploy other technologies as guardrails for AI, instead of turning to unreliable human oversight, Blumofe suggests.

“You need non-AI systems in the guardrail role,” he explains. “These technology tools would help to automate testing and validation of AI outputs, flag issues, and have the capability to pause the AI work. This keeps humans out of approval loops, while also helping to reduce risk.”

When humans aren’t the right choice

Other IT leaders suggest that human-in-the-loop systems aren’t the right solution in every AI use case. When AI is used to flag and mitigate cybersecurity incidents, for example, waiting for a human to approve an action may be too late.

“If an endpoint is compromised, you may want the system to isolate it immediately,” says AISquared’s Kimura. “Waiting 20 or 30 minutes for someone to approve that action could allow the attack to spread.”

The objective is not to put a human into every AI decision, he adds. “It is to put the right human, with the right context and authority, at the right point in the workflow.”

AI is already changing how software is built. Google Cloud’s DORA research, based on nearly 5,000 technology professionals, found that 90% now use AI at work, spending a median of two hours a day with it, which translates to roughly a quarter of the working day.

In many organizations, the focus is on what happens at the end of the lifecycle: how much code is generated, how many steps are automated, how quickly code is shipped. While those are visible signals of progress, they can be divorced from actual value. Google’s DORA research found that while AI adoption lifts delivery throughput, it also increases instability: more software shipped less predictably.

The more impactful change is happening earlier in the lifecycle. Requirements, design and test strategy shape everything that follows. When those stages are structured correctly, downstream execution becomes faster, more consistent and easier to control. When they are not, issues tend to carry through the entire system, regardless of how much automation is applied later.

The Standish Group’s CHAOS research has consistently put insufficient user involvement and incomplete and changing requirements at the top of the list of reasons projects fail, with only around 31% delivered on time, within the budget and matching the intended scope.

Generalist models are good at producing plausible early-stage work, but can fall flat when outcomes are measured holistically. Setting off in the wrong direction can have lasting consequences. It carries through design, into code, into test cases written against the same flawed assumption.

This is why building AI into software engineering is less about adding tools to existing workflows and more a wholescale reconsideration of the product development lifecycle.

Building narrow agents into the lifecycle

The most effective approach is to break the product development lifecycle into modular agents with narrow scope: one converts discovery material into structured requirements, another produces technical design, the other generates and runs test strategies. Narrow scope keeps each agent’s context manageable and its output consistent.

Importantly, this creates clear points of control. At each stage, AI proposes and progresses the work, while human roles review, challenge and approve before it moves forward. As a result, features can move from discovery to production-ready code far faster than before: design cycles compress, and test scripting that took four engineers can be handled by one, freeing up time for higher value work.

Those checkpoints matter because plausible output is the hardest kind to catch. Stack Overflow’s 2025 survey found 66% of developers name “AI solutions that are almost right, but not quite” as their single biggest frustration, and 45% say “debugging AI-generated code is more time-consuming.”

Without a review gate at each stage, that cost compounds rather than surfacing. GitClear’s  AI Code Quality research shows the trade-off more clearly: refactoring line moves are down 70%, and long-term legacy maintenance is down 74% versus 2022 levels, yet copy-paste, code block duplication and other indicators of technical debt continue to rise.

Governance calibrated to risk

None of this is safe without governance designed in from the first step and calibrated to risk. In practice, that means deploying agents in read-only mode before they are given authority to act. It means setting confidence thresholds before any routing decision is automated. This requires human sign-off on novel exception types even after an agent has proven reliable, and keeping a full audit trail across every decision point.

Much of the market is not there yet. The Cambridge Centre for Alternative Finance’s 2026 Global AI in Finance Services report found 78% of regulators rate explainability as critical or important to their objectives, while only around half of industry firms have adopted explainable AI methods. That gap illustrates how governance expectations continue to outpace implementation.

This discipline runs in two directions. We hold ourselves to it internally, in how we engineer, because anything we build for a regulated market has to survive that scrutiny first. It also must hold in the client’s environment: the firms we build for answer to regulators for every automated decision, so governance cannot be bolted on at the end – it needs be present at every step.

Clients in regulated markets need determinism and explainability. A system that runs end-to-end without a traceable, governed path is hard to put into production, however well it performs in a demo.

Measure the outcome, not the output

Counting volume is easy: more agents, more generated code, more automated steps feel like demonstrable progress. The metrics that matter include quality, real-world outcomes and cost to build.

One example: building connectors between Xceptor and third-party platforms through a conventional engineering process could take around two weeks. Running the same build through the AI-native product development lifecycle – agents generating requirements, design documentation, code and test strategies, with engineers reviewing and steering at each stage rather than producing from scratch – took two days. For clients, that difference means integrations stop being a bottleneck on go-live. Total cost to build also fell 83 per cent, including AI token spend.

Another example is the first agent we built for financial institutions, focused on extracting data from trade confirmations. Firms are often managing large volumes of confirmations which arrive in unstructured formats across document types, such as emails, PDFs and SWIFT messages – and extracting this data is where AI agents excel, delivering significant efficiency and accuracy gains.

From doing to directing

Building an AI-native product development lifecycle changes what engineering work looks like. As agents absorb repeatable execution, the human work concentrates on judgement: architecture, edge cases and steering output rather than generating it.

We found that after a short time, our engineers were no longer producing first drafts; they were reviewing and refining agent output. Sometimes they corrected the outputs, but more and more they were able to approve what was generated. The cognitive load moved from production to verification. This shift from making to directing and validating is the clearest sign of a maturing AI-native engineering model.

Eventually, we will think less as fixed teams and more as cells – product roles and builder roles working alongside AI, each person operating above the task they used to own. The role of a QA Engineer will shift towards creating the paved roads and guardrails that humans and agents use, enabling quality to be built in consistently across every cell.

It would be dishonest to frame this only as acceleration. When work you have done for years becomes something you direct rather than do, that is a real adjustment, and leaders who pretend otherwise may lose their best people to organizations that manage the transition better. Mandating tools is not the same as helping people use them well; in our experience it produces more licenses installed, not more work changed. Adoption comes from champions, role-specific playbooks and measuring delivery outcomes: a people-first approach rather than a procurement one.

None of this works without both sides. True AI-native product development depends on continual, close collaboration between humans and machines. Years of domain knowledge, paired with the speed and pattern-recognition of these systems, is what makes the outcomes better, not the technology on its own. That combination is what makes the process repeatable at scale.

Every agent you add multiplies your output, but it also multiplies the surface area your governance has to cover. The question every CIO should be asking is how to review at your new output volume without lowering your standards. The answer is to be deliberate about what a person looks at and what the system can handle on its own.

Done well, this is not a tradeoff between speed and rigor. Your standards hold on everything an agent produces, and the people reviewing spend their attention where it earns the most. Getting there comes down to how you sort the work and where the checks live.

The review bottleneck that limits scaling agents

Start with the verification, since it can be written down. Encoding the checks you already trust means expressing them as rules the pipeline runs, tied to the work itself rather than to a reviewer’s memory: Tests have to pass, security scans have to run clean, and the change has to trace back to an approved task. Once those live in the system, it runs them on every change an agent makes without getting tired.

Then let the risk of each change decide what happens next. Picture an agent creating a change in the system: The checks run the moment it does, and one that passes cleanly and touches nothing sensitive can merge on that basis. A change that alters permissions or reaches into a critical path takes a different path, holding until the named owner of that area signs off. Nothing sensitive slips through on a green check alone, and nothing routine sits in a queue waiting for attention it doesn’t need.

Set that up once, and every change an agent produces meets the same bar. People still look at the ones that need a human eye.

Your governance layer is invisible to the agents doing the work

Rules only scale if they run inside the system agents already work in, not in a separate tool a person has to open. A layer like that can only report after the fact. It sits outside the path the agent travels, so nothing enforces it in the moment, and no reviewer can keep pace at agent speed. Put the rules in the path the work already travels and no one has to remember to enforce them: An agent can’t route around them, and a reviewer doesn’t have to go looking. In practice, that looks like a few things happening on their own:

• A change that touches critical/sensitive systems routes to the required approver before it can move forward.
• Work that isn’t tied to an approved task never merges in the first place.
• The audit trail builds itself as the work happens, instead of being pieced together afterward.

Handled this way, governance is built into how the agents work.

Scale agents without scaling risk

At real volume, the payoff comes from putting your standard into the system itself. Approvals then go to the human judgment calls, and everything else clears against deterministic rules that never skip a step. For a CIO, the question worth asking is whether your standards exist in a form the system can act on, or only in people’s heads and in documents no agent can make sense of.

It also changes what governance is worth to you. When your standards run inside the system, adding more agents makes the work more consistent, because every one of them is held to the same encoded bar. With the routine standards running on their own, your engineers are free to spend judgment where it counts, and governance keeps pace with the work rather than trailing behind it. It becomes the thing that lets you scale up with confidence.

Visit jira.dev to see how leading organizations keep their people on the judgment calls and let the system run the routine checks, so their governance runs automatically.

Over the past few years, much of the conversation has focused on autonomous AI and how quickly organizations can remove humans from decision-making. In financial services, we’re seeing the opposite trend. The organizations making the most sustainable progress aren’t eliminating human oversight—they’re redesigning it.

The model taking hold within the banking industry isn’t AI that operates independently and makes decisions; it’s AI that operates with intent and oversight. Human-in-the-loop is quickly becoming the standard, combining the speed and scale of machine-driven insight with the accountability, judgment and control that organizations can’t afford to lose. The shift is increasingly aligned with how regulators and industry frameworks are shaping responsible AI adoption, from the NIST AI Risk Management Framework to the revised U.S. banking agencies’ model risk management guidance, both of which reinforce governance, monitoring and accountability over blind automation.

From my perspective, this is not innovation slowing down; it’s AI adoption growing up. The first wave of enthusiasm focused heavily on what could be automated, but now the more important question is where can AI create meaningful value while keeping the right human judgment, oversight and accountability in place? In financial services, that distinction matters. In an industry built on trust, those capabilities are not optional— they are foundational to how we serve customers, manage risk and earn confidence every day.

Banking offers one of the clearest examples of why human-in-the-loop AI is becoming the default operating model for enterprise AI more broadly. Some of the most valuable AI use cases sit in environments where mistakes carry real consequences, customer impacts are significant and explainability is essential. In those moments, human oversight is what allows institutions to scale AI responsibly.

In banking, AI usually doesn’t operate in a vacuum. Whether it supports customer service, fraud detection, compliance, underwriting or internal productivity, it is touching workflows that affect customers, colleagues, regulators and the reputation of the institution. That is why responsible scale matters. Global bodies including the Financial Stability Board and the Bank for International Settlements have recognized the efficiency and analytical benefits AI can bring, while also warning that it can amplify model, cyber, concentration and governance risks if controls do not keep pace. For financial institutions, the mandate is clear – move with ambition, but scale with discipline.

Where human oversight matters most

The next phase of enterprise AI adoption will be defined by how well institutions understand where AI can move work faster, and where human judgment still needs to lead. For financial institutions, that starts with materiality. The greater the potential impact on customers, regulatory obligations or financial resilience, the stronger the case for meaningful human oversight.

Customer service is a good example. AI can help teams summarize inquiries, recommend next-best actions and reduce manual handling time. But when the issue involves a disputed transaction, a vulnerable customer, a complaint or product suitability, human judgment must remain central. AI can make service faster. It can make it more consistent. But it cannot replace empathy, context or accountability.

Fraud and financial crime are areas where AI can create real value, but human oversight remains essential. AI can detect patterns, anomalies and suspicious behavior across large data sets at a speed and scale people cannot match, but fraud is dynamic. Typologies evolve, bad actors adapt quickly and authorities have warned that AI can also increase the sophistication of scams, fraud and disinformation. In that environment, analysts and investigators play a critical role — validating signals, reducing false positives, escalating the right cases and applying judgment as the threat landscape changes.

Risk, compliance and credit are similar. AI can help synthesize internal data, identify control gaps and strengthen monitoring. But when outcomes affect lending decisions, regulatory obligations, capital or liquidity, institutions need governance that preserves challenge, review and accountability. The EU AI Act’s human oversight requirements for high-risk systems point to a broader direction of travel — the more consequential the use case, the more important it is that people can understand the system’s limitations, override outputs and intervene when needed. For U.S. institutions, the specific rule may differ, but the principle is already part of how banking operates. High-impact decisions require accountable oversight.

At the same time, human-in-the-loop cannot mean putting a manual checkpoint in front of every AI-assisted task, which would slow adoption and reduce the value AI can create. The goal is risk-based oversight. Lower-risk use cases may be managed through periodic review, testing and monitoring, while higher-risk applications may require real-time review before action is taken. What matters is that institutions define those thresholds clearly, rather than assuming one oversight model fits every use case.

Why collaborative AI is winning in financial services

The financial institutions that are embracing human-in-the-loop AI do so because they understand both the opportunity and the stakes. AI can process transactions, summarize complex information and identify patterns at a scale humans cannot match. At the same time, consumer expectations make clear that scale alone is not enough.  TD Bank’s 2026 AI Insights Report found that 78% of Americans now use AI-powered tools in their daily lives, yet only 18% are comfortable allowing AI to make important financial decisions independently. That gap says a lot about where the market is heading. Consumers are not rejecting AI, but they are drawing a clear line around accountability.

That is why speed cannot be the only measure of success. When customer outcomes, regulatory obligations or enterprise risk are involved, people still need to challenge the output, apply context and remain accountable for the decision.

That oversight matters because AI does not always fail in obvious or familiar ways. Generative AI can produce confident but inaccurate answers. Machine learning models can drift as data changes. Even highly accurate systems can deliver biased or poorly reasoned outputs when the data, assumptions or prompts behind them are flawed. NIST’s Generative AI Profile highlights risks including confabulation, privacy concerns, misalignment and automation bias.

For financial institutions, the lesson is that responsible AI requires people who understand how to use the technology, and also when to question it.

Building the organization for responsible AI at scale

In addition to being a technology challenge, responsible AI is also an operating model challenge. The institutions that scale AI well tend to do three things with discipline: establish clear governance, redesign workflows around the technology and build the skills employees need to use AI responsibly.

Governance starts with ownership, but it cannot sit with one executive or one team alone. It requires coordination across business lines, risk, compliance, legal, technology and model risk functions. That cross-functional model is becoming more common as organizations move beyond experimentation.  McKinsey’s State of AI report found that AI governance is often jointly owned and that CEO involvement in governance is correlated with stronger reported bottom-line impact, suggesting that firms derive more value when AI oversight is treated as an enterprise priority rather than a side initiative.

Workflow redesign is just as important because the real value of AI comes from reimagining processes end-to-end. That means identifying where AI can handle summarization, pattern recognition or drafting and where people should focus on exception handling, complex decisions and relationship-driven work. Human-in-the-loop is not about preserving the old operating model; it’s about building a better one.

That also requires new capabilities across the workforce — employees need to know how to use AI tools effectively and how to challenge them. They need to understand prompt quality, output limitations, data handling expectations and the warning signs that a system may be producing unreliable results. The  World Economic Forum’s 2025 report on AI in financial services underscores that while adoption is accelerating, responsible scaling depends on workforce adaptation, governance maturity and a clear understanding of risks alongside value creation. Responsible adoption depends as much on human capability as it does on model performance.

Looking ahead, enterprise AI in financial services will become more embedded, more specialized and more agentic in targeted domains. But that does not mean the human role becomes less important; if anything, it becomes more important. As AI takes on more analytical and operational work, people will increasingly serve as orchestrators, reviewers and decision-makers at the points that matter most. They will set objectives, define controls, interpret edge cases and know whether the AI results can be relied upon.

The organizations that lead in AI will be the ones that make human judgment a deliberate part of the design—clear about where AI can accelerate work, where people must remain accountable and how both can operate together with discipline. For financial institutions, the call to action is to treat human-in-the-loop AI as the operating model that makes innovation more trusted, more durable and more worthy of the customers and communities it serves.

In July, two frontier labs disclosed cases in which cyber-capable agents crossed the intended boundaries of evaluation environments and reached real production systems at external, unrelated organizations. Most commentary since has focused on which company a court would find liable. There is a more immediate concern for anyone operating agents in production, and it’s not about the law. If this happened in your deployment tomorrow, who would bear responsibility for the incident?

I have a particular purpose for stating it that way. This spring, I reviewed the Coalition for Secure AI’s Shared Responsibility Framework before its publication in May. Frameworks like that, along with the cloud shared responsibility models that preceded them, break down responsibilities among the parties operating a system: provider, platform, developer, deployer, user. July illustrated what happens when the entity suffering the damage is none of the above. Responsibility maps stop at contractual boundaries. Agent reach does not. Call it the reachability gap.

The two disclosures described different failure modes, and that difference is significant. OpenAI was testing models against a cyber benchmark with production refusals reduced so the evaluation could measure real capability. The models obtained internet access through a zero-day in a package registry component, went looking for the benchmark’s answer key and inferred that Hugging Face might host it. Hugging Face reconstructed the intrusion from over 17,000 recorded agent events, and the campaign extended further than initially disclosed, affecting accounts on four external services.

Anthropic’s incidents were not escapes. In a review of more than 141,000 cybersecurity evaluations, Anthropic found three cases in which Claude models reached the internet from inside or alongside a third-party evaluation environment and then accessed real systems at three organizations. Live connectivity was mistakenly available. The models had been told in their prompts that they had none, and Anthropic stated that Claude did not exfiltrate itself or deliberately attempt to escape its test environment. Of the affected organizations the lab was able to reach, two had not detected the activity before being notified.

The pattern held while I was writing this. On August 4, OpenAI disclosed two more incidents from third-party evaluations, separate from Hugging Face. In one, a partner running capture-the-flag exercises had a testing environment misconfigured with live internet access, and the fictional target in the exercise happened to share its name with a real domain. The model exploited an actual website, taking it for part of the simulation, then found and used credentials to operate it. Whoever owned that site had no relationship with OpenAI, with the evaluation partner or with the test. They were reachable, and their name collided with a fiction.

Count the parties. Two frontier labs. A third-party evaluation partner. A platform victim. A cloud customer victim. Organizations that learned of a breach from a notification. CSO has already examined how the response strained the AI tooling defenders had available. One step earlier: who bears responsibility for reaching for that tooling at all? At the moment of detection, who had ownership of containment?

One accountable party per activity

The Shared Responsibility Framework’s core rule is almost boring to state: for every activity across the AI stack, there should be exactly one accountable party. This principle breaks down the AI deployment process into five layers, from business usage to model supply chain, and maps eight roles across them. This way, detection, containment and remediation each carry a name before an incident rather than during one. This rule exists due to the failure it prevents, and the framework explicitly names it: in the absence of accountable parties, teams default to finger-pointing. The model provider blames configuration. The platform points at the tenant. The application team cites model limitations. Everyone is partially correct, and the clock continues to run.

July reads differently than the coverage suggests when measured against that rule. OpenAI, as a model provider, evaluation platform operator and agent-deploying organization, took on at least three roles. Having multiple roles within one company is common and not necessarily dangerous. The danger appears when those roles are not broken down into distinct internal owners and decision rights handoffs. When provider, operator and deployer are one and the same and those lines are not drawn, the question of which function failed has one answer and thus no answer of value.

Anthropic’s incidents make a related point from the opposite direction. There was a boundary, this time, between the lab and its evaluation partner, and the incident resided in a gap between two different understandings of what the environment allowed. The public disclosures do not clarify how responsibility was shared contractually regarding each incident, and I will not speculate. What is clear is that whatever boundary existed failed to provide a common understanding of one important control, which is whether the evaluation environment was allowed to access the internet. A boundary that has never been tested against a question that simple is not a boundary at all. Both sets of incidents also occurred in situations of high autonomy where the standard production safeguards were reduced, and the models were allowed wide latitude to pursue an open-ended goal, which is the context where unclear boundaries lead to the greatest losses. I am not suggesting that either lab acted carelessly. Both institutions made rapid public disclosure of details, and both have named the conditions that allowed the activity. That is the point. If the most knowledgeable and incentivized parties ended up with no clear answer on whose incident it is, we should not presume to have one.

The victims were outside everyone’s map

Here is the part the framework does not resolve, and I say that as one of its reviewers. Shared responsibility models, CoSAI’s included, presume a value chain. Each role is taken up by a party who voluntarily entered into the relationship. This is what makes responsibility assignable via contracts and review boards. Hugging Face did not take up any of those roles. It was neither a customer, nor a vendor, nor an evaluator of OpenAI. The same goes for the three organizations that Claude reached. They were just reachable.

That’s the structural lesson I’d put on one slide for a leadership team. In classical models of cloud shared responsibility, the provider-customer boundary was visible because the boundary was contractual. With agentic systems, there is another kind of gap, the reachability gap: an agent’s reachable range may stretch to actors outside of the deployment relationship, and there is no contract that articulates what obligations you owe them if things go sideways. Any mapping exercise that ends at the edge of your value chain is tackling the easier part and skipping the part July was about.

What to do before it is your incident

None of this requires buying anything.

• Your map has to look beyond your value chain. For each agent, identify what it can access that you have no agreements with: public infrastructure, other tenants, the open internet. A long list next to broad latitude shows your true exposure, regardless of what your contracts stipulate.
•  Determine who is authorized and obliged to inform an outside party that your agent might have affected. This is the gap July most starkly revealed, and this is not a question you want to bring up during an incident when the answer involves a lawyer, a communications person or an executive who has never considered it.
•  Establish accountability by specific component and activity rather than by organizational box. For each production agent, specify who owns detection, containment, eradication, recovery, remediation, then slot those assignments into the five layers. Where one team or vendor occupies multiple roles, capture the consolidation and identify the internal handoffs, because an undocumented internal boundary will break under pressure.
• The contracts for your most autonomous agents are worth retrieving. If a vendor’s agent is able to operate across domains within your environment, review whether any provision assigns responsibility for the consequences of those actions. In the contracts I have seen, the answer is typically silence, and silence is a decision someone else will make for you later.
• Treat evaluation and red-team environments as production-impacting systems. Their safeguards are often reduced on purpose, and that is exactly why containment, monitoring, incident command and external notification procedures around them should be at least as rigorous as those protecting production systems.

The questions about liability will likely be tested in court or by regulators, and those answers will be about the labs and their evaluation partners. The operational question is already yours. Hugging Face learned whose incident it was from forensics. Two of Anthropic’s victims learned from a notification. The organizations that come through the next one intact will be the ones that were able to answer the question before it was asked.

There are few professions where part of the job is literally to read the newspaper, watch television or continuously check what is happening across every channel. In a media organization, that is not confined to journalists. Keeping up with events is, in one way or another, part of the work of much of the organization. A newsroom is also one of the most paradoxical professional environments I know. Before moving into media, I was director of New Technologies at Spain’s leading franchise consultancy, a role that allowed me to work with companies of almost every size and across almost every sector.

After years as CIO of Canarias7, I have seen how some of the most senior journalists approach almost anything related to technology with suspicion while looking back fondly on an idealized newsroom of typewriters, cigarettes and whiskey. It may sound like a caricature, but there is truth behind the nostalgia. Few professions have gone through so many technological transformations while continuing to defend the essence of their craft.

That is precisely why I believe CIOs have a great deal to learn from newsrooms in the age of artificial intelligence. Not because journalism is an example of frictionless technology adoption, but because newsrooms have spent decades solving a problem that is now spreading across the enterprise. They absorb huge volumes of information, distinguish what matters from what does not, make decisions with incomplete data, change priorities within minutes, verify before acting and keep accountability for the outcome clear.

As AI evolves from assistants that generate information into agents that can recommend, decide and execute actions inside enterprise systems, the challenge is no longer purely technological. It is also an operating model problem. That is where the experience of a newsroom becomes particularly relevant to a CIO.

Human attention should be treated as a scarce resource

In a newsroom, it is impossible to subject every piece of information to the same level of review. The credibility of the source, the relevance of the story, the consequences of a possible error and the existence of conflicting accounts all influence how much effort is devoted to verification. Some routine decisions are made quickly, while others pass through several levels of checking before publication.

Companies are beginning to face the same problem with AI. The most common response has been to place a person at the end of the process, the familiar human-in-the-loop, to validate what the machine does. But if AI multiplies the number of analyses, recommendations and decisions, it can also multiply the workload of those expected to supervise them. Human attention, just as in a newsroom, has to be allocated according to risk.

CIOs should apply the same principle. A routine, reversible, low-impact action can be automated, while an anomalous or low-confidence decision involving sensitive information or consequences that are difficult to reverse should escalate to a higher level of supervision. A 2026 study of automation bias in computational pathology involving 28 specialists found that in 7% of cases, an initially correct assessment was overturned after the specialist received an incorrect AI recommendation. Simply adding a person to the process does not automatically remove AI risk.

The Associated Press offers a useful example of this logic. Its updated AI standards from July 2026 allow AI to assist with early-stage research, transcription, translation, document summaries and headline suggestions, while keeping editorial judgment, verification and accountability with journalists. The point is not to require human supervision for everything, but to reserve it for tasks where context, interpretation or the consequences of an error justify that control. For a CIO, that means determining which decisions can be automated and which should escalate to a person, based on risk, impact, and reversibility.

Governance must be built into the architecture, not the prompt

A newsroom does not control its work through instructions alone. Reporters, writers and editors have different responsibilities and decision rights and the workflow itself establishes points where a story can move forward, be sent back for further verification or be stopped. Control depends not only on each person remembering what they are allowed to do, but also on how far their authority extends.

The Replit incident in July 2025 showed why the same principle matters in AI systems. Jason Lemkin, founder of SaaStr, was using Replit’s coding agent to build an application when the agent deleted a production database despite explicit instructions not to make changes during a code freeze. The agent had direct access to the same database used by the live application. Replit subsequently introduced automatic separation between development and production databases so that changes made during development could not directly affect live customer data. What a written instruction in a prompt failed to prevent became constrained by the architecture itself.

For a CIO, the parallel with a newsroom is straightforward. A reporter can prepare a story without having the final say over whether it is published. In the same way, an agent can analyze data or recommend an action without necessarily having permission to execute it. The system that generates a transaction should not always be the same one that validates it. Separating functions, limiting permissions, distinguishing between read and write access, isolating environments and maintaining audit and rollback mechanisms translate into AI architecture a principle that newsrooms have applied for decades: the person or system proposing an action does not have to be the one with authority to approve and execute it.

Humans need power, not simply a place in the process

Human-in-the-loop has another limitation. A person can be inside the process without having meaningful control over it. Researcher Madeleine Clare Elish coined the term moral crumple zone to describe situations in which responsibility for the failure of an automated system is attributed to a human operator who had limited control over what happened.

An “Approve” button does not guarantee effective supervision if the person clicking it does not understand why the system is recommending an action, does not have time to review the evidence or lacks the authority to stop it. In a newsroom, an editor can challenge a source, demand additional verification, delay publication or stop it altogether. The editor is not simply part of the process but has authority over it. That is why I find it more useful to think in terms of a human-in-power model, where responsibility, context and veto authority remain connected.

This logic also applies to organizational design. When a major story breaks, a newsroom first tries to understand what is happening and then decides how to cover it on the website, in print or on social media. Something has changed in recent years: the story comes before the channel. Many companies still operate the other way around. Information and processes remain fragmented across CRM, ERP, finance, marketing, customer service and data platforms. If we introduce autonomous agents into each silo, we can increase speed without ensuring that all of them are acting on the same version of reality.

Data, identity, context, permissions and traceability should therefore become shared capabilities on which different AI systems operate. Otherwise, automating faster may simply mean producing inconsistencies faster.

There is one final characteristic of newsrooms that I find particularly relevant. A plan agreed first thing in the morning can be obsolete ten minutes later. At Canarias7, we hold daily meetings with the different teams involved, from journalists and homepage editors to SEO, analytics, multimedia and other support functions, to review what is happening, what has changed since the last decision, which stories require more attention and where resources should be concentrated. When something important happens, priorities change, teams reorganize and decisions are made again. This is not a failure of planning. It is a normal condition of the job.

AI is taking companies toward the same environment of continuous decision-making. As systems absorb more information and agents gain more autonomy, they will need to update their context and reconsider actions when reality changes. The operating model has to make that possible without losing accountability along the way.

After years of working between journalism and technology, I still find it paradoxical that a profession that so often greets each new tool with skepticism can now offer some of the best clues for organizing the AI-driven enterprise. Artificial intelligence can accelerate our ability to observe and act, but the advantage will lie in what happens between those two points: how we interpret information, set priorities, verify what matters and determine who has the authority to decide. In that sense, the CIO in the AI era is becoming the editor of how the organization observes the world, makes decisions and acts.

Every Friday, Bret Greenstein, CAIO at consulting firm West Monroe, holds a company-wide meeting to share what’s happened in AI over the past week. He also spotlights one employee at the firm who’s created their own AI agent from the ground up, which lives in the company’s internal AI store. Since the store launched in May, more than 200 employees across departments — many without any technical, engineering, or coding background — have created over 550 agents.

“About 15% of our firm builds all the time now,” Greenstein says. “That’s a huge population.”

Enabling employees to spin out their own agents has become popular at many firms. Staff have built hundreds of agents at software company Blackline, for instance, and Microsoft has deployed more than 500,000 internal agents to help employees streamline workflows. Gartner also anticipates that by 2028, global average Fortune 500 companies will have more than 150,000 agents.

Employees know the intricacies of their work, the biggest pain points, and time drainers, so they can build solutions that address those specific issues, according to Greenstein. It also creates enthusiasm, empowers employees, and fosters innovation among the workforce as they build from the ground up.

That said, there’s been a pivot over the last six months, says Michael Murphy, partner and AI practice lead at global management consulting firm Adaptovate. When agentic AI first came on the scene, companies went all in, pushing to build and agentify nearly anything they could. In recent months, however, the narrative has shifted to getting a handle on agent sprawl, assessing the value agents deliver, and keeping costs in check.

“We’re really at this interesting inflection point where clients are having to figure out if we built the right agents, and are they delivering the value we expected,” Murphy says.

Today, tech leaders face a three-way squeeze, says Tiago Azevedo, CIO at AI-powered low-code development platform OutSystems. From the workforce side, many employees ask for permission to use more AI, but the CFO says token usage is becoming too big an expense on the balance sheet, and the CEO wants to see innovation and results from workforces using AI agents.

“I think that’s the biggest challenge for a CIO,” Azevedo says. “Let people take advantage of the technology but in a way that’s cost-effective and actually brings ROI.”

Building in a controlled environment

Employees have built myriad tools to aid their daily workflows. Azevedo’s company launched an agent dubbed Signal Sam, which searches databases of prospective customers, and gives account executives information to pitch them. Murphy and Greenstein also mention finance departments using agents to scan and categorize invoices, HR conducting a first pass on résumé screenings via agents, legal teams utilizing a self-service agent for NDAs, and marketing employees building agents that pull and analyze data from CRMs. These tools are often created by non-technical employees who’ve never written a line of code.

With so many agents popping up, CIOs need a way to oversee them, and ensure they meet corporate standards but without choking innovation, Azevedo says.

He recommends role-based access controls embedded into tools and configured behind the scenes. “So we allow them to use, but in a way that’s governed and controlled, because that’s our duty to the organization,” he says.

Ivan Burazin, CEO and co-founder of open-source developer platform Daytona, advises CIOs to treat agents like employees. “You’re not going to bump into them in your local Starbucks,” he says, “but you give them tasks and they have access.”

So set up agents with specific credentials, like how an organization would grant access to a new hire, with a laptop locked down with organization security protocols, Burazin adds. He also recommends sandboxing, in which agents operate in isolated machines with scoped credentials and firewalls so the sandbox prevents agents from accessing corporate systems or data outside allowed perimeters.

Organizations could use an internal ticketing system as well where employees wanting to build agents request a new identity for them, Burazin says. That way, tech leaders maintain visibility and governance over new agents.

“If something goes haywire in audit logs tomorrow, you can see it’s that agent versus an actual human,” Burazin continues.

He acknowledges that giving employees what feels like free rein to build and run agents can induce stress for CIOs and CISOs. But if a company doesn’t proactively establish tools, employees are apt to privately build AI in the shadows. As long as agent development happens within established confines, it won’t create problems organization wide.

“If you just enforce the security posture that you would for humans, you’ll save yourself a lot of headaches,” Burazin says.

When creating the AI store, Greenstein started by certifying tools for chat, code, data analysis, and other tasks, and then trained employees and made the tools broadly available to use. That process created guardrails and an inherently secure building environment. It also allows tech leaders to continue to monitor prompts and activity.

Now, tech teams review what’s been built in the AI store and flag any agents that excel. If employees have built 10 project management tools, for example, the leader will tag what they deem the best one. That gives employees the option to use existing agents or build a separate version for themselves.

More agents, more tokens

Over the last three to six months, Azevedo has been hearing from customers that their biggest hurdle is agent sprawl and the increasing cost those agents bear due to token usage.

In mid-July, OpenAI published a guide around useful work per dollar, sharing how leaders can look at tasks completed, time saved, and decisions improved to determine if their AI investments are bearing fruit. In addition to using the guide, Murphy suggests comparing the labor time and cost to conduct a manual task against time saved by using an agent, including which type of model the agent requires.

A cheap flash model, for instance, could be easy to justify the cost. “If it’s a very expensive Opus or Fable level model, that’s going to be a lot more challenging of a cost equation,” Murphy says. He adds that making this comparison isn’t about replacing the workforce but swapping “knucklehead admin work” for more engaging, human-centric work. This change may also require some organizational restructuring, such as CIOs and HR leaders working more collaboratively to handle change management as job responsibilities shift. Without the workforce optimized to work with agents, organizations won’t see the promised ROI of use cases, Murphy says.

West Monroe also informs its employees on the costs of different models. Without knowledge about tokens and costs, many employees defaulted to the highest-end model for any tasks before understanding that models come with different price tags. “We started educating people on the various relative costs of different models, and they immediately adjusted behavior, and our cost dropped,” Greenstein says.

While strictly quantitative returns are one way to measure ROI, Greenstein also thinks about return in a qualitative sense. “What does speed get me?” he asks. If someone in the firm is able to follow up with a client in hours because of an agent’s assistance, rather than days or weeks without one, the client will be impressed, and the firm might win their business over a competitor.

“Tokens will cost money no matter what,” he says. “But if you maximize the return, it’ll far outweigh the cost.”

Organizations are making one of the biggest decisions about AI without realizing it.

Every time we automate a process…

Every time we deploy a generative AI copilot…

Every time we trust an AI recommendation…

We’re answering a question that most leadership teams have never actually discussed.

Who — or what — should be making the decision?

Over the past year, almost every conversation I’ve had with executive peers has eventually turned to AI. The questions are usually the same: How are you using it? Where are you seeing value? How fast should we move?

They are all good questions. But I think they are causing us to skip a much more important one: Where should AI make decisions, where should it advise and where should human experience and judgment always lead?

I’ve spent my career helping organizations navigate major technology shifts: EPR, cloud and analytics all helped people make better decisions.

AI feels different.

For the first time, technology isn’t just helping us make decisions. It’s beginning to participate in them.

In many cases, it already can. The better question is whether it should. Because every time AI makes a decision, we’re making one too.

We’re deciding which decisions belong with AI, and which still belong with people.

As organizations move beyond experimenting with generative AI, the challenge is no longer deploying it. It’s redefining how people with AI work together. Microsoft’s Work Trend Index describes this shift as organizations move from experimenting with AI to fundamentally changing how work gets done.

What I learned was that we were deciding where AI should participate in making decisions.

That realization led me to what I now call the decision line.

Drawing the decision line

I don’t think organizations need another AI framework. What I learned is they need a better way to think about where AI belongs. That’s what the decision line is: A simple way to think about where AI should decide, where it should advise and where human judgment and experience should lead.

Most organizations are putting formal AI governance in place, such as the NIST AI Risk Management Framework. But every leadership team still has to decide where the AI should participate in business decisions. A framework can help organizations manage AI risk, but it can’t determine where AI belongs in your business.

Not every decision deserves the same level of human involvement.

Some decisions are routine, repeatable and governed by well-defined business rules. They’re decisions where consistency matters more than interpretation and where manually reviewing every transaction doesn’t create additional value.

One of the first places I saw this play out was in Accounts Payable. Organizations have long relied on people to perform three-way matching and resolve exceptions between purchase orders, invoices and goods receipts. Most transactions follow established business rules, making them ideal candidates for AI.

When we automated that process, AI performed the routine work, allowing people to focus on the exceptions that actually required experience and judgment. The result wasn’t just greater efficiency — it was a better use of people’s expertise.

I saw the same thing happen in Logistics. AI evaluated transportation costs, truck capacity, inventory, purchase orders and delivery schedules in seconds, allowing planners to spend more time making decisions instead of running calculations.

The question isn’t whether AI can process more information than people. The better question is whether human judgment materially improves the outcome.

In these situations, it often doesn’t.

Those decisions naturally belong below the decision line.

The harder question — and the one I think every organization will wrestle with — is what belongs at the decision line.

 Where the decision line really matters

The decisions below the decision line are usually the easiest to identify. The hard decisions are the ones where AI is incredibly valuable, but human judgment still changes the outcome.

The moment this really became clear to me was when we started using AI to answer two important questions about new restaurant locations.

The first was relatively straightforward: Where should we open the next one?

To answer the first question, we built an AI model to analyze sales cannibalization. We tested it against historical data, creating what I called a “time machine” to see how accurately the model could predict outcomes we already knew.

The model consistently predicted sales cannibalization more accurately than the old spreadsheet models we relied on for years.

What I learned was that AI could solve a business problem we had struggled to model for a while.

Then we asked the second question: Where should we open the next restaurant?

The model evaluated demographics, population density, traffic patterns, household characteristics and every criterion we provided. On paper, many of its recommendations looked exactly right.

Then something interesting happened.

Our franchise office started to challenge several of the recommended locations.

Not because they disagreed with the data. Because they knew something the data couldn’t.

They understood that customers don’t always behave the way models predict. Some customers had been going to the same restaurant for more than 5 years. Even when another location opened closer to home, many continued to go where they felt comfortable. They knew the quality of the food. They recognized the wait staff. They had established routines. Convenience wasn’t always measured in miles.

The model also couldn’t recognize what experienced Franchise operators noticed immediately. One location had excellent demographics but poor visibility from the street. Another was difficult to enter because of traffic patterns and an awkward parking lot.

One neighborhood looked ideal because it was busy during the workweek. Most of that activity, however, came from nearby businesses during lunchtime. On weekends, the area became surprisingly quiet.

None of those realities existed in the data.

The AI wasn’t wrong. It simply didn’t have the context. That wasn’t a failure of the model. It was a reminder that some business knowledge isn’t captured in historical data. It lives in the experience of people who know the customers, understand the operations and recognize what data alone can’t measure.

What I learned was that the real question wasn’t whether AI was right or wrong. It was where AI belonged in the decision-making process. That’s the decision line.

That’s where AI creates the most value. AI contributes to the analysis. People contribute the context.

Together, they produce a better decision than either could have made alone.

Every organization will draw the decision line differently

One of the biggest lessons I’ve learned is that there isn’t a universal decision line. Every organization has its own business model, customers, operating priorities and confidence in its data, so every organization should draw the line differently.

I also don’t think the decision line is permanent.

As organizations improve the quality of their data, strengthen their business processes and gain confidence in AI, the line will naturally move.

Decisions that require human involvement today may become routine tomorrow.

That’s the process.

But I also believe some decisions will always require people.

Not because AI isn’t capable. Because some decisions require accountability, context and judgment that extend beyond what data alone can provide.

The goal isn’t to move as many decisions as possible to AI.

The goal is to decide intentionally where AI creates the most value and where human judgment and experience create the greatest impact.

It’s also a conversation many CIOs are having as AI governance moves from theory to day-to-day leadership, a topic CIO.com has explored in its coverage of AI governance.

Formal AI governance frameworks are valuable, but they don’t answer an important question: Where should AI participate in your decision-making process?

That’s a business decision. And it’s one that every leadership team has to answer for itself.

Before implementing AI at scale, I ask four questions:

1. Which decisions are truly routine and repeatable?
2. Where does human judgment materially improve the outcome?
3. Where should accountability always remain with people?
4. What would need to change before we move another decision below the decision line?

I’ve found those questions often lead to better conversations than asking where AI can be used. They shift the discussion from technology to business value.

When I think back to the conversations I’ve had over the past year, I’ve realized the biggest question isn’t how quickly organizations adopt AI. It’s where AI belongs.

Some decisions clearly belong with AI. Other decisions clearly belong with people.

Most organizations will spend the next several years deciding everything in between. Every Organization implementing AI is making one of the biggest decisions about AI.

Make sure it’s one you’ve made intentionally.

That’s the decision line.

A fraud model looks at an insurance claim and returns a score of 0.23: low risk. A SHAP explanation lays out exactly why: no prior claims, a modest claim value, an unremarkable claimant profile. A human adjuster reads the explanation, agrees with it and signs off. Every box that explainable AI asks us to check has been checked. The claim is settled and closed.

By any current standard for responsible AI, this is exactly what a good outcome looks like.

Two months later, an audit

Two months after the claim closes, it’s pulled into a routine quarterly audit sample (no suspicion attached, just standard sampling). The auditor isn’t reviewing the model’s reasoning. She’s asking a narrower question: can this decision be reconstructed, independently, from its own evidence trail?

She doesn’t start with the fraud score. She starts with the paperwork.

Her first question: what established the date of the incident? She pulls the document meant to corroborate it and checks something the claim file itself never surfaces, the document’s own creation metadata. It was generated two days after the claim was filed. She asks the obvious follow-up: did anyone check this before the claim reached the model? Nobody had. Document provenance was never captured anywhere in the intake pipeline as a fact to be verified; the claim reached the model looking exactly like every clean claim before it.

Knowledge gap: This doesn’t establish that the claim is fraudulent. A prior version of the document may exist and say the same thing, in which case the late timestamp is a non-event. What it establishes is that the system has no way to answer that question either way. Document lineage was never captured, so there’s no way to know whether an earlier version existed, or whether it agreed with this one.

Her second question: when was the insured asset last valued? The figure the model used is 22 months old relative to the date of loss. Nothing about it is wrong or falsified; it’s simply the most recent valuation on file, resolved to the wrong point in time and handed to the model as though it were current.

Knowledge gap: This doesn’t establish that the valuation is wrong. The asset’s value may not have moved materially in 22 months, in which case the old figure and a fresh one would say the same thing. What it establishes is that the system has no process for knowing which is true. There’s no trigger for re-valuation and no record of whether the figure was ever checked against current conditions.

Her third question: as of what date was the claimant’s risk profile evaluated? The “current risk” features the model consumed were pulled at query time (the day the claim happened to be scored) rather than resolved back to the date of loss. She asks whether that was a deliberate design choice. Nobody can tell her. There’s no rule on record, no document, no named owner for the decision “risk features should reflect the date of loss, not the day the system happens to be queried.” It’s simply how the pipeline was built.

Policy version gap: This doesn’t establish that the score would have come out differently under incident-time data. The claimant’s risk profile may not have changed since the date of loss, in which case query-time and incident-time features would produce the same result. What it establishes is that nobody decided which point in time should govern the decision, so there’s no way to know whether the two agree, only that nobody checked.

Three questions. Three gaps. Not one of them touches the model, and not one of them would have shown up in a model audit, but all of them question the explainability of the decision taken.

The blind spot in explainability

This is the uncomfortable pattern showing up across banking, insurance, healthcare and government AI deployments: the model is explainable, the explanation is faithful and the decision is still wrong. It’s worth being precise about why, because the instinct is to reach for a better interpretability technique, or a smarter document-reading agent, and neither fixes this class of failure.

Every mainstream explainability approach shares one architectural trait, whether it’s SHAP, LIME, DARPA’s XAI program, the EU AI Act’s Article 13 disclosures, or IBM’s AI FactSheets: they all operate after the model has reasoned. They take the inputs as given, assumed correct and explain what the model did with them. That’s legitimate and valuable. It’s also, structurally, a blind spot for the class of failure the audit just surfaced.

An enterprise decision isn’t just a model output. It’s a reasoning process running over a knowledge base and a policy set, both of which have to be current, correctly sourced and resolved to the right moment in time before the model ever sees them. A model can reason perfectly over compromised inputs and produce a perfectly explainable, perfectly wrong decision. No amount of post-hoc transparency catches that, because post-hoc transparency was never looking at the inputs; it was only looking at what the model did with them.

Put precisely: a decision is explainable only when its reasoning, its knowledge and its policy are all explainable at the same time. Explainable AI delivers exactly one of those three: explainable reasoning. It can be flawless, and the decision can still be unexplainable overall, because an unverifiable knowledge input or a misapplied policy breaks the chain regardless of how good the reasoning layer is. That’s the precise sense in which explainable AI is necessary but not sufficient. Sufficiency is a property of the whole decision (knowledge, policy and reasoning together), not of any one component, however mature that component’s tooling has become.

Where the audit’s three questions actually point

Map the auditor’s three findings back onto that structure and the pattern sharpens.

Her first and second questions (the backdated document, the stale valuation) are both knowledge failures. Neither is a data-entry error; both are facts that were never traceable to an authoritative source, versioned and resolved to the right point in time before reaching the model.

Her third question (features pulled at query time rather than incident time) is a policy failure. No rule existed to say which moment in time the decision should be evaluated against, so the system defaulted to whatever was easiest to compute: now.

Governing the inputs, not just the output

The fix isn’t a better explanation of the model, and it isn’t a more sophisticated agent reading the documents more carefully. It’s governance that sits before the model, a framework I call Explainable Decision Systems (EDS), built on three pillars plus a temporal layer that ties them together.

This tracks a broader shift already visible across enterprise AI deployments: the stacks that win aren’t the ones that ship fastest, but the ones that get data and decisions governed before anything downstream runs.

• Explainable Knowledge (EK). Every fact fed into a decision has to be traceable to an authoritative source, versioned and (critically) resolved to the valid time of the event being decided, not the time someone happened to query the system or the date typed into a form field. An asset valuation from 22 months ago isn’t wrong data; it’s data resolved to the wrong point in time.
• Explainable Policy (EP). The policy version that governs a decision is the one that was active when the event occurred, not the one currently in the system. A claim from November has to be judged against the policy wording in force in November (exclusion schedule and all), retrieved and locked before reasoning starts, not looked up after the fact.
• Explainable Reasoning (ER). This is where existing XAI actually earns its keep, but EDS splits it into two accountable parts: AI reasoning (EAI), which needs interpretability techniques like SHAP, and Human Decision (HD) (overrides, approvals, escalations), which needs an identified actor, a timestamp and a documented justification. EDS doesn’t claim human judgment is formally explainable in the way a model’s math is; it just insists that it’s attributable and auditable.

The layer that binds all three together is bi-temporal governance: knowledge and policy have to resolve to the real-world moment the decision concerns, not the moment the system happens to be queried. In EDS, this resolution (called a temporal witness) isn’t a report generated after the decision, the way the audit in this piece was. It’s a gate the decision has to pass before reasoning is allowed to start. Had the witness been in place, the claim in this scenario would never have reached the model at all; it would have been flagged as a governance exception the moment the document’s creation timestamp failed to match its claimed valid time.

Why no AI agent can catch the hospital record

This is worth slowing down on, because it’s the clearest illustration of why EK has to be its own dimension, entirely separate from model explainability or agentic sophistication.

Every clinical or administrative record system has two different dates living in it. There’s the stated field value (“date of incident,” “date of visit”), which is just a form entry: whatever a person typed, told a clinician or was asked to record. And there’s the system-of-record creation timestamp: the moment the backend actually inserted that row, logged that transaction or wrote that file. That timestamp sits in the origin system’s audit trail, not on the document itself.

A claimant doesn’t need to forge anything to exploit the gap between those two dates. They need only have a record created on 30 November whose stated incident-date field reads “15 November.” The document is completely genuine. Nothing about it is altered, and no forensic check (pixel analysis, formatting consistency, metadata inspection of the file itself) will find anything wrong, because there is nothing wrong with the document. The lie isn’t in the artifact. It’s in the mismatch between what the artifact claims and when an entirely separate system actually created it.

Hand that document to an AI agent, no matter how advanced, and ask it to check for fraud. It will read the stated date, assess whether the narrative is internally consistent, run every forensic technique available and come back clean; it’s answering the wrong question. It’s checking whether the document is coherent and unaltered. The question that actually matters is whether the document’s claimed date matches the creation-time metadata logged by the system that produced it. That answer doesn’t live in the document. It lives in a separate system, under separate access control and requires a deliberate governance step to retrieve and cross-reference. No amount of model capability substitutes for that step, because the information the model would need simply isn’t present in what it’s being asked to read.

This is exactly the gap enterprise surveys keep flagging. In Grant Thornton’s 2026 AI Impact Survey of nearly 1,000 senior business leaders, 78% said they lacked confidence their organization could pass an independent AI governance audit within ninety days, precisely because most are scaling AI they cannot explain, measure or defend. That’s the downstream version of the same problem this claim ran into: explainability treated as a reporting exercise rather than a precondition the case has to clear before it ever reaches the model.

That is precisely what Explainable Knowledge formalizes: a requirement that every input be provenance-linked to its origin system, with independently retrieved creation-time metadata checked against the claimed valid time; not a soft appeal to “make sure your data is good,” but a specific, checkable technical control. Under EDS, if that cross-reference can’t be performed, the claim doesn’t proceed to the fraud model. It’s blocked as a governance exception, regardless of how convincing the document looks or how confidently any reader, human or AI, assesses it.

These aren’t academic distinctions, either. In the UK and international context, failures like these map to live regulatory obligations: FCA claims-handling rules on treating customers fairly, Federal Reserve SR 11-7 guidance on model risk management, and the EU AI Act’s Article 9 risk-management requirements for high-risk systems. Regulators are already asking questions that XAI, and the agents built on top of it, cannot answer.

What this means for the people building these systems

For enterprise architects, the practical shift is this: stop treating knowledge and policy governance as documentation you produce after a model runs (or worse, as something an audit uncovers two months later) and start treating it as a gate the model has to pass through before it runs. This is the same direction major risk-consulting practices are now pushing clients toward: governance embedded as a control layer across the AI lifecycle, rather than a compliance sign-off at the end of it. Concretely, that means:

• Bi-temporal data stores for anything that feeds a decision, so “valid at the time of the event” is a queryable property, not an assumption.
• Provenance links from every document or claim to its origin system’s own creation-time metadata, captured independently of the stated field values on the document (not derived from them).
• Policy version registries with effective and expiry dates, so the version applied is probably the one that was active when the event occurred.
• A pre-inference middleware layer that checks provenance, versioning and temporal validity before forwarding anything to the model, and blocks or flags the case if it can’t.
• Explicit, attributable records for human overrides: who, when, under what authority and why.

None of this replaces SHAP, LIME or your existing model-risk documentation, and it isn’t a call for a smarter document-reading agent either. It sits in front of both, deciding whether they’re even looking at trustworthy, independently verified inputs.

The bar has moved

Explainability was always going to be necessary. Regulators demand it, customers expect it and no serious enterprise AI deployment should ship without it. But “the model is explainable” and “the decision is governable” have quietly become two different claims, and treating them as one is how a technically pristine 0.23 fraud score sails through, gets closed and sits untouched for two months until an auditor asks the one question explainability was never built to answer: not what did the model do, but what was it allowed to look at, and when, and against what independent record?

Architectures that can’t answer that question will keep producing decisions that are perfectly explainable and quietly wrong, right up until the audit finds them.

Let’s say one of your teams builds an AI agent that actually works. Word gets around, and seemingly overnight, there are twenty more built by people in finance, legal, HR, and support. Most of them are useful, but when someone suddenly gets a chatbot response showing customer data they shouldn’t have access to, reality hits.

The real test of enterprise AI readiness isn’t at all whether your coworkers can confidently work with AI. Instead, it has everything to do with governance and security, global, cross-cutting policy, and privacy. Many enterprises are underprepared to face these issues: a survey from Databricks and the Economist found that “40% of respondents believed their organization’s AI governance program is insufficient.” And Microsoft’s Data Security Index reports that “only 47% of organizations across industries report they are implementing specific GenAI security controls.”

Having worked with many CIOs to develop strategies to govern their AI systems, this piece is a deep dive into the specifics of what works and what doesn’t. Keeping employees, customers and your entire organization safe must be your top priority before you even start rolling agents out.

8 layers of governance every enterprise needs

It’s no doubt that the enthusiasm for AI is real, but so is the list of questions that bubble up a month later: Who’s allowed to publish an agent to the rest of the company? How do we track versions, and can we roll one back? Can we require SSO on every agent? What data does this have access to, and does it respect the permissions on those documents? Where do the logs live, and how quickly can we see them?

I think of enterprise AI governance as a set of layers, each answering one of the questions above. You can build them incrementally, but ideally all eight are in place before you have more agents than you can list off from memory.

1. Roles and groups. Whatever platform you choose, ensure that it offers granular role-based access control. At the level of abstraction higher, map groups to real departments (Legal, HR, Capture Team) and assign those groups to their own. Keep the admin count small enough to list out loud. Though this is the coarsest measure, it still bears a big responsibility down the line — according to Fortune Business Insights, “RBAC solutions help reduce unauthorized access incidents by nearly 30%, enhancing data security.”
2. Scope. It’s extremely helpful to build out private folders with explicit allowlists, so a project, agent, or workflow isn’t merely locked to outsiders but invisible to them. Agents with access to sensitive customer data shouldn’t advertise their own existence.
3. Change control. Those who are building AI agents should be able to place locks on so that only the owner can edit it, with admin override. Every change should land in a version history with a diff and a commit message, and rolling back should take one click so that you know exactly what was edited if an agent stops performing. According to Google’s DORA State of DevOps research, teams with strong version control and rollback practices recover from failed changes in under an hour, versus a week or more for teams without them.
4. Publication. When a builder finalizes an agent and wants to release it to the rest of their team, depending on the use case, they should consider adding one-click SSO, a password option for external collaborators, and restrictions on which origins and which users can reach it. OWASP’s 2025 Top 10 keeps broken access control at #1, observing that every application they tested had some form of broken access control.
5. Org-wide policy. This is the layer CIOs underuse. Require SSO on all interfaces rather than hoping. Restrict who can publish, so shipping to the company is an admin action. Create approval workflows, where a builder requests review and an admin does the publishing (similar to traditional SDLC best practices around pull requests). Allow or deny specific tools and connectors across the enterprise, so nobody connects a data source that procurement hasn’t approved yet. Lastly, vary policy by group, so only Legal can reach the legal agents.
6. Data access. Obsidian Security’s 2025 AI Agent Security Landscape report found that 90% of deployed AI agents are over-permissioned relative to the actual scope of their assigned tasks, and separately estimates that agents are typically granted about 10 times more access than their workflows need. That’s why connections and knowledge bases deserve their own permission model. Credentials should be encrypted and owned by whoever created the connection to a given tool, app, or data lake, with sharing as an explicit decision. Here’s where it gets tricky: if a builder connects SharePoint with their own account, the agent sees what that person can see. Connect through a service account and the agent inherits the service account’s entire scope, which is usually far wider than anyone intended. The safer pattern is to check end-user access at runtime, so the person asking has to authenticate before retrieval happens and only gets what they’re already entitled to.
7. Observability. It’s imperative to have exportable records of who ran what, when, against which model, with token counts and latency, plus per-step traces showing inputs, retrieved chunks and outputs. Give builders the ability to mask or disable logging where the data is too sensitive to retain. Then push those records out of the platform on a schedule so security’s pipeline consumes them automatically.
8. Authentication. SSO over passwords, MFA through whatever authenticator your org already runs, and defaulting new users into the lowest-privilege role until an admin promotes them. According to Microsoft, MFA can block more than 99.2% of account compromise attacks. And SSO gives you the email address of everyone using your interfaces, which makes for an additional layer of security.

None of these are new inventions; they’re the same controls you already apply to internal software. But most organizations haven’t extended them to AI that both talks to customers and writes to the CRM.

Don’t forget about deployment

After governance, the other half of enterprise AI readiness is deployment. Regardless of what platform you choose, there are four realistic postures:

1. Multi-tenant SaaS, where you share infrastructure with others and rely on tenant isolation at the database layer.
2. Dedicated single-tenant, where the vendor operates a VPC that only you occupy.
3. Bring-your-own-cloud, where the software runs inside your own cloud account and your data never leaves it.
4. On-premise, where you own the infrastructure, the control plane and the data plane outright.

It’s well known that multi-tenant is available as soon as possible and updates itself continuously. Dedicated takes a couple of weeks to stand up and updates on a schedule the vendor applies. Bring-your-own-cloud runs two to four weeks and splits responsibility, with the vendor managing the application through a scoped cross-account role while your team owns networking and IAM. On-premise starts at a month or more, and after that your team applies every release itself.

My honest read? Most organizations are well served by multi-tenant, and choosing it doesn’t compromise on security or privacy. Bring-your-own-cloud and on-premise make sense when you have a data sovereignty requirement and a mature internal platform team, but they result in manual updates and slower time to value. There’s no right or wrong answer here — but it’s helpful for CIOs to understand the trade-offs between each of these common deployment methods so they can choose what fits best for their enterprise.

A readiness test you can run this week

So, are you ready for enterprise AI? Here’s a quick self-diagnostic that I like to run with CIOs. Pick your three highest-privilege AI agents or workflows in production and try to list these attributes yourself:

Who owns each one by name. What changed in it most recently, and who approved that change. Which connectors and knowledge bases it can reach, and whose permissions those reads run under. Whether accessing it requires SSO. Where its logs are, and how long they’re retained. What happens to it if the model version it depends on is deprecated next quarter.

If most of those answers are “I’d have to ask around,” you’re not ready to scale, and that’s fine. All it means is that you have a week of policy work ahead — but you’ll thank yourself later for figuring the tough stuff out first.

For many organizations, AI is proving easy to pilot but difficult to scale. Pilots often look inexpensive because they run on narrow datasets with a handful of users, explains Ben Schein, chief AI and analytics officer at cloud software company Domo. “But the cost lives in deployment, the moment you connect that capability to real workflows and the systems of record behind them,” he says. “That’s when the real bill appears.” So CIOs must always budget for the gap between when it works in a demo and when it produces governed and durable value.

Organizations can easily get caught out because they run pilots as a technology experiment instead of a business initiative, he adds. “The interesting question is never whether AI can do the thing in a demo,” he says. “It’s whether it should run in this process, and whether it survives contact with production.”

There’s also a lot of pressure on IT teams to be doing something with AI simply because everyone else is, says Naren Gangavarapu, chief transformation and AI officer at Australian Cruise Group.

He calls it AI theater because there’s a big show around AI even though there aren’t that many successful applications of the technology in production environments.

AI costs out of control

According to John D’Emic, CTO at AI observability platform Revenium, one of the big traps when running a pilot is failing to anticipate how quickly consumption can spiral as adoption grows. “As an example from our own engineering org, back in May, a developer opened an AI coding session on his laptop, and it stayed open for four days,” he says. “By the time it closed, it had run 4,819 calls and cost us $3,762. We didn’t budget for this, and no alert fired. But that one session cost more than a lot of teams spend on their entire monthly AI tooling.”

While this showcases how a developer can make a costly error, Dmitriy Anderson, CIO and digital and social commerce leader at home and gardening retailer Leroy Merlin South Africa, believes the pilot trap frequently happens when employees with little or no software development experience vibe code applications. “It doesn’t matter if you can create something in 15 or 20 minutes if the result is AI slop,” he says. “Think dirty code, no consideration for safety, security, and possible data exposure.” In most cases, these pilots are developed with one of the frontier apps, and someone probably used their personal AI subscription, so the costs are negligible, he adds. But if you have a company of several thousand people, and you now want to roll this tool out more broadly, that’s where costs can get out of control.

This scenario is only exacerbated by the introduction of agentic AI, D’Emic adds. “Agents don’t spend money at human speed,” he says. “In the old cloud days, an engineer could spin up infrastructure in minutes and finance might not see the bill for a month, which was painful but recoverable. Agents, though, call APIs around the clock without waiting on anyone’s approval.”

Mind the trap

While cost is a big factor in the AI pilot trap, it should be treated as a symptom of a bigger problem, says Schein. The underlying issue is governance and observability. “An autonomous workflow can fan out into more queries, API calls, and model invocations than anyone scoped,” he says. “So if you can’t see what it’s doing, and spend compounds quietly, you only find out once the invoice arrives.”

In a recent LinkedIn post, Anderson outlined how in just six weeks he built a platform for a fraction of the sticker cost using three AI models orchestrated together. The traditional estimate to build the same tool would have required 2,472 engineering hours from a team, and was expected to take around nine months. “I went through the proper engineering steps and planning, and made sure the application passed a series of cybersecurity frameworks,” he says. “The purpose of this exercise was to showcase that AI can still speed up the process even if you take the time to work through the necessary steps. You can build with AI rigorously and securely.”

So to turn AI experiments into enterprise value, every AI interaction must be attributable: who triggered it, against what data, on which model, and at what cost, Schein says. For each workload, be sure to ask how often it runs, which model tier the job actually needs, and what triggers it, human or automatic. “A frontier model on an automatic trigger and a small model called on demand are completely different cost curves for the same task,” Schein adds.

For Anderson, it’s helpful to use AI to highlight potential gaps, assumptions, or blind spots in your ideas early on. “When you start building an idea, ask the agent to interview you,” he says. “It will go through every phase and ask questions about the important facets of the process, from scalability and budget to deployment options. You can even make AI write a prompt for itself, because it knows its capabilities and quirks better than you ever will. It’s called meta prompting.”

Anil Inamdar, global head of data services for the Instaclustr BU at NetApp, suggests CIOs cost out the whole program, not just the demo. “Generally, the model itself is the cheapest part of the program,” he says. For him, it’s important to have security and governance people in the scoping meeting, not the launch meeting.

He believes the pilot trap is also, or perhaps mostly, a sequencing trap. “A lot of teams are wired to build first and ask permission later, only to discover months down the line they can’t pass a security review or data privacy audit without a painful and costly rebuild. It’s also valuable to define what failure looks like before you define success.

“Pilots tend to die because of no result, which isn’t the same as a bad result,” Inamdar says. “Emphasize to the deployment team on day one that if a target result by a certain month isn’t seen, we shut it down. Otherwise, you’re funding a zombie pilot because everyone’s invested and no one wants to be the one to call it out.”

A few months ago, I was sitting in a glass-walled conference room with the executive team of a fast-growing enterprise. The vice president of customer operations was enthusiastically demonstrating the new automated agent features their software vendor had just pushed into their CRM platform.

On the screen, the software looked brilliant. The agent could read customer complaints, analyze transaction histories and automatically resolve issues. The VP showed us how the system could independently offer retention incentives to unhappy accounts without a human ever touching a keyboard.

Then I asked a simple question: “What is your approval process when the AI decides to grant a $20,000 contract discount to keep a customer from leaving?”

The room went completely silent. The VP looked at the director of IT, the director of IT looked at the chief risk officer, and everyone realized the same thing at the exact same moment. They had spent three months evaluating software licenses and security protocols, but nobody had asked who gave the software permission to sign off on corporate spending.

Major software providers like Salesforce, SAP and Oracle are rapidly moving beyond simple report writers and conversational chatbots. They are embedding active, autonomous agents directly into the transactional core of systems that manage your revenue, customer agreements and financial ledgers. According to Gartner’s latest adoption forecasts, eighty percent of enterprise applications will deploy these embedded capabilities by 2026. These applications do not just summarize data: they issue refunds, alter contract terms and trigger supply chain orders.

When I review these deployments with client teams, the core problem has nothing to do with artificial intelligence. It is a fundamental breakdown in corporate delegation and signing authority.

The breakdown of the corporate signing matrix

Every mature company I work with operates on a clear delegation of authority matrix. This framework dictates exactly who can sign off on financial commitments. A vice president might have authorization to approve spending up to $500,000, a director might sit at $100,000 and a front-line manager might be capped at $500. For two decades, technology leaders have spent millions of dollars building security and compliance controls to ensure every human employee operates strictly within those limits.

Yet when a software vendor releases an update featuring autonomous agents, companies routinely grant these features unrestricted operational freedom. Because the capability arrives as a native feature inside an existing application, business units enable it with a single click. In my advisory work, I repeatedly see organizations grant third-party software features more financial freedom than their own human managers.

This represents a massive blind spot in executive governance. McKinsey’s global surveys on artificial intelligence reveal a striking pattern across the enterprise landscape: while adoption is accelerating at a historic pace, only a tiny fraction of organizations are actively managing the financial and operational risks of automated decision errors.

The quiet cost of shadow delegation

In my audits, this rarely manifests as a dramatic system crash. It plays out as a quiet margin leak. In one organization I reviewed, a department head had enabled an automated customer retention feature over a weekend. The agent noticed an important account expressing frustration in a support ticket, and to prevent the account from churning, it independently applied an unapproved 15 percent discount to their multi-year contract.

The customer was happy, and the account manager considered the client saved. But from an executive perspective, an unvetted third-party algorithm just executed an unauthorized contract modification that eroded company margins. When the finance team conducted a quarterly audit, they did not discover an employee violating spending policy. They discovered a black-box automated decision that bypassed every internal approval control in the company.

When an auditor tests your internal controls, presenting a log showing that a vendor’s algorithm made an unauthorized financial change does not satisfy the requirement. If an action requires managerial sign-off when performed by a human being, letting software execute it independently is a major control failure.

How I advise executive teams to handle automated authority

Protecting your organization does not mean turning off these tools or falling behind on technology. It means treating vendor-supplied agents exactly like third-party contractors who have not yet passed a background check.

Forrester Research emphasizes that extending zero-trust security frameworks to automated business processes is now mandatory for enterprise risk management. Zero-trust simply means that no user, device or automated tool gets implicit trust. Every proposed action must be validated against explicit business rules before it happens.

When I help enterprise teams design these safeguards, we establish a practical three-tiered boundary for automated tools:

• Read and draft permission: Automated tools can freely analyze trends, draft emails and assemble internal reports. No human sign-off is needed to create a draft, but the system cannot publish or execute anything on its own.
• Standard administrative permission: Tools can handle routine administrative tasks or process standard requests below a strict financial cap (such as a $50 service credit), provided every single action is logged in an audit file that managers review weekly.
• Restricted financial permission: Any action that alters contract terms, changes pricing tiers or issues major refunds are strictly held in an authorization queue. The system generates the request, but a human manager must click “approve” before the change hits the live database.

As a technology executive, you cannot control what automated features software providers bundle into their platforms. You can, however, control the financial boundaries and signing authority those tools are permitted to exercise within your business.

What to do at your next executive leadership meeting

1. Ask for an automated authority inventory: Have your team audit your core software platforms to identify every automated feature currently running with permission to alter financial or customer records.
2. Revert to draft-only mode: Instruct your team to default all vendor-supplied automated agents to “draft only” until a clear business case justifies giving them independent operational authority.
3. Establish a firm human-in-the-loop rule: Require a strict organizational policy that no automated system can modify pricing, contracts or financial ledgers without explicit manager approval.

Speed-to-market dominates enterprise AI priorities in 2026. Beyond upfront resourcing costs of prioritizing speed, organizations face a more insidious risk:  the compounding cost of ungoverned AI.

In November 2019, a tech entrepreneur signing up for the newly launched Apple Card publicly complained that he received a credit limit 20 times higher than his wife’s, despite joint tax filings and her higher credit score. Steve Wozniak had a similar experience: a limit 10 times higher than his wife’s. Retrospectively, these revelations were the canary in the coal mine.

In the years that followed, Apple and its credit partner, Goldman Sachs, drew legal and regulatory scrutiny over gender bias and consumer protection issues. The CFPB’s 2024 order documented that Apple had forced Goldman Sachs to accelerate deployment by attaching a $25 million penalty to every 90-day launch delay. 

Prioritizing launch speed — ship first, address problems later — over building a functioning disputes process created years of cascading failures. Apple and Goldman Sachs were ordered to pay $89 million in penalties and consumer redress. Prohibited from launching another credit card until it could demonstrate a credible plan to comply with the law, Goldman Sachs lost money on Apple Card for years and ultimately sold its consumer credit line. The legal and compliance penalties were only a fraction of the total costs.

If a deterministic underwriting system can create liability at this scale, the risks posed by agentic AI are substantially greater:  autonomous systems can multiply and scale errors, quietly and invisibly, at machine speed.

The accelerated cost of ungoverned speed

In software engineering, shortcuts taken to ship are called “technical debt.” When teams sacrifice robust architecture, processes or solutions to reach deadlines, interest accrues in the codebase as brittle integrations and expensive refactoring. When it comes to agentic AI, technical debt accrues faster. AI portfolio returns are estimated to drop by 18% to 29% when technical debt is ignored.

Similar to technical debt, AI governance debt accumulates when speed-to-market routinely takes precedence. Unlike technical debt, which can wait silently in a repository without immediate consequence, governance debt is neither patient nor pausable. Deployed without clear authorizations, boundaries, or constraints, AI systems scale defects across an enterprise at machine speed.

The agentic multiplier: Scaling decisions and outcomes

When a standard LLM produces an output, a person receives it and decides what to do with it. That pause is a crucial point of control:  a “human gate” stands between the generative AI model and the consequence. AI agents operate in continuous, (semi-)autonomous loops, without human gate friction. With agents, a model error can cascade downstream unimpeded through enterprise systems. Multi-agent systems inadequately governed have error rates of nearly 20%. Generative AI scales outputs, and agentic AI scales outcomes. Put another way, agentic AI scales outcome-producing actions, and every autonomous action carries a decision that a human used to make. 

AI governance is often mischaracterized as “putting the brakes on” speed-to-market. In practice, omitting it causes “velocity decay.” While 54% of leaders consider governance to be an obstacle to scaling, its absence or inadequacy creates sociotechnical bottlenecks that ultimately stall deployment and operations. To prevent both velocity decay and governance debt, governance must “shift left” to be architected throughout the AI system lifecycle.

Accountability reduces the debt

Technical accountability within an autonomous system cannot exist in a vacuum; it requires both structural and cultural accountability throughout an organization.

Structural accountability

Structural accountability assigns formal ownership over agent actions to specific human decision-makers. A July 2026 white paper, Safeguards for Agentic Finance at Runtime (SAFR), provides specific case studies from the financial services industry that include and advocate for structured human accountability.

• Individual ownership of agent decisions:  Any decisions made by an agent that are highly consequential require a human owner. Think of a CFO signing her name to financial statements: if auditors or regulators take issue with those documents, she is named as responsible for the numbers. Her successor inherits the same accountability. Consequential decisions, such as a denied insurance claim, a moved financial asset or a rejected job applicant, require a human owner and a process to identify them (detailed below).
• Shared ownership of agent outcomes:  When an autonomous workflow crosses traditional siloes, such as a logistics agent altering supply chain routes based on real-time marketing data, ownership becomes distributed. While externally, an organization remains a single liable entity,  internally, it must assign accountability across organizational boundaries. Mechanisms like a Joint Accountability Agreement (JAA) can facilitate this by explicitly aligning cross-functional decision rights, escalation protocols and continuous monitoring metrics. 

Cultural accountability

Cultural accountability means everyone has a role to play, and everyone owns both the final result and the process to get there. Think of a crew team rowing: everyone rows to win, and everyone is responsible for both individual performance (like erg times) and the team’s overall success (race speed and ranking). High-ownership cultures ensure that accountable behaviors are recognized and rewarded, and visible consequences exist when accountability is lacking. Just as accountability is instilled in a crew team through clear, shared goals and transparency on individual and team effort, metrics and results, employees can be incentivized to own individual and collective actions, outputs and outcomes. Importantly, these accountable behaviors enable accountable AI. If these employees are also empowered to challenge AI, they are equally empowered to own its results. Knowing they will be rewarded or recognized for interceding – not punished – is critical to reduce governance debt and prevent velocity decay.

Human accountability in the agent workflow

Product Advisory Collective

The diagram maps structural and cultural human accountability with an agent’s workflow:  monitor context, make decisions, coordinate, complete tasks, and deliver an outcome. When decision-making is consequential, a single human owner must be accountable (Step 2), whereas shared ownership of agent outcomes, across all of the teams and individuals that contributed to the workflow or are impacted by its results (Step 5), is typically necessary. Underpinning all five steps is cultural accountability, which facilitates accountable human behaviors and enables human workers to detect and prevent unaccountable agent behaviors.

Importantly, whether an agent’s decisions (Step 2) require a single named owner depends on the severity of consequences for the enterprise and its key stakeholders. A named owner signs off on the risk criteria and thresholds, and answers for any consequences that occur if risk thresholds are surpassed. The SAFR white paper recommends evaluating five risk criteria: action reversibility, financial materiality, customer impact severity, regulatory sensitivity and novelty or anomaly. Risk thresholds are set pre-deployment, and proposed agent actions (Step 2) are evaluated continuously at runtime. While “above threshold” risks trigger a real-time human-in-the-loop (HITL) review, the executive owner remains ultimately responsible for any resulting repercussions or systemic issues.

Too often, model decision-making oversight is lacking. While agent orchestration and escalations to HITL reviewers are established practices, scrutiny over the risk or consequences of the model’s decisions is the exception, not the rule, even in high-risk industries. As recently as 2023, roughly 40% of hospital systems did not evaluate AI models for accuracy and 56% did not evaluate them for bias. This clear lack of oversight is governance debt with acute liability. Preventing and reducing the debt requires formal structural ownership and a culture that rewards accountability. 

Strategic recommendations

A unique organizational impact of agentic AI is that it collapses traditional operational boundaries, like safety and security, converging performance metrics, systemic risks and multi-jurisdictional compliance requirements over time. To address this convergence of performance and governance issues, we recommend systemic interventions, including that leadership:

• Streamline governance efforts strategically to account for the collapsed operational boundaries and eliminate organizational friction and bottlenecks.
• Structure accountability into roles:  assign names to consequential decisions and set up JAAs to manage shared accountability.
• Shift governance left by embedding oversight early and continuously across the system lifecycle.
• Design governance ex ante, not ex post: consider it a core architectural and infrastructural requirement, rather than post-launch remediation.
• Scale efficiently with policy-as-code: supplement or replace siloed and unnecessary manual oversight with automated computational controls.

Final takeaways

For CIOs, the ultimate mandate is to anchor AI governance directly in day-to-day infrastructure, roles and responsibilities. When autonomous agents scale risks instantly, traditional manual reviews and human-controlled workflows simply cannot keep pace. Oversight must “shift left” and be strategically built into system logic from the start.

Unchecked speed-to-market is an expensive illusion. As Apple Card’s costly errors and losses demonstrate, a rushed launch and a lack of oversight can result in financial penalties, remediation and loss of trust, reputation and business. To avoid the repercussions – including the velocity decay – that accompany governance debt, leaders must proactively architect accountability across the enterprise, the system lifecycle and critical decisions.

Eliminating systemic blind spots doesn’t require perfect foresight; it requires intentional architecture, well-designed collaborative ownership across silos and explicitly named human decision-makers. It’s possible to prevent the compounding liability that falls between AI decision-making and unstructured human accountability by answering two key questions early on:  Who owns which decisions? And how do we clearly incentivize accountability?

We are all transforming now.

Some companies have formally named transformation programs. Others are being transformed by a new regulation, an AI mandate, a cyber event, a weather disruption, a change in customer behavior, a competitor’s move or an urgent demand to reduce costs.

The label is almost beside the point. The operating assumptions keep changing, and the company has to change with them.

Accenture’s Change Reinvented research found that 95% of organizations had undergone at least two transformations in three years, while only 30% of C-suite leaders expressed confidence in their organizations’ change capabilities.

More recently, a McKinsey Global Survey of more than 1,200 executives and managers found that 40% expect their current business models to require significant change within three years simply to remain economically viable.

Transformation is no longer an event that temporarily interrupts normal operations. It is becoming normal operations.

That changes the role AI is beginning to play. We are not using it only to draft emails, summarize documents or write code. We are increasingly asking it to interpret complex situations, identify options, recommend priorities and influence consequential business decisions.

I believe that can be enormously valuable. I also believe it requires a boundary we have not defined clearly enough.

AI should help us understand reality. It should not be allowed to negotiate with it.

The impossible request

I have sat in versions of this meeting many times.

The company must reduce spending by 10%. A regulatory deadline cannot move. The CEO has declared AI a strategic priority. A customer initiative has already been promised to the market. Hiring is frozen. Several of the same architects, data specialists, cybersecurity professionals and change leaders are required by every program.

Each commitment may be rational. Together, they may be impossible.

Someone asks the AI assistant to recommend a plan that protects all of them.

The answer arrives almost immediately. It proposes phased delivery, tighter governance, selective automation, resource sharing, increased collaboration and a revised sequence. It sounds balanced. It may even sound reassuring.

But did the answer prove that the commitments can coexist? Or did it produce the most plausible story that satisfies the request?

That distinction matters.

In 2025, OpenAI rolled back an update to GPT-4o after concluding that the model had become overly flattering and agreeable. OpenAI described some of the responses as overly supportive but disingenuous and acknowledged that the model had been too influenced by short-term user feedback.

The point is not that AI cannot be trusted. The point is that an AI system can be highly intelligent, useful and well-intentioned while still being pulled toward the answer its user would prefer.

That is manageable when the stakes are wording or tone. It becomes dangerous when the question is whether the enterprise can afford, staff, sequence and deliver everything leadership wants.

A separation of powers

Organizations need a separation of powers for AI-assisted decision-making.

Human judgment should establish intent. Leaders decide what matters, which outcomes deserve protection, what risks are acceptable and which tradeoffs the organization is willing to make. No mathematical model can decide what a company ought to value.

Governance should establish authority. It determines who may change a priority, move a date, redirect capital, accept more risk or relax a constraint.

PMI’s 2026 Closing the Change-Readiness Gap report argues that enterprise agility depends on aligning intent, authority, structure and trust. Yet only 41% of executives surveyed believe their operating models support rapid allocation of capital and talent.

The ability to move resources quickly is important. So is the ability to see what that movement changes elsewhere.

Mathematics should establish feasibility. Once the assumptions, capacity, funding, dates, dependencies and constraints have been made explicit, the organization needs a protected calculation of whether its commitments can coexist.

Math does not decide strategy. It does not make imperfect data perfect. It does not remove politics, judgment or uncertainty.

It does establish where judgment ends and wishful thinking begins.

If 12 initiatives need the same six specialists during the same quarter, the organization does not have a communication problem. It has a capacity collision.

If a budget reduction removes the resources needed to achieve the original business case, the economics have changed even if the presentation has not.

If two regulatory commitments depend on the same release window, confidence will not resolve the sequence.

AI should establish understanding. It can question assumptions, find inconsistencies, identify patterns, propose alternatives and explain why an option succeeded or failed. It can help leaders ask better questions and explore complexity without waiting for days of manual analysis.

But the sequence matters.

AI may recommend that an assumption change. It should not silently change that assumption to produce a more acceptable answer.

Protecting the truth layer

CIOs are accustomed to protecting data. The next challenge is protecting the authority of different kinds of information.

A recorded fact is not the same as an assumption. An approved risk tolerance is not the same as an executive preference. A mathematically calculated shortfall is not the same as an AI-generated interpretation. A proposed option is not a commitment.

When all of these appear in one polished response, the distinctions can disappear.

EY’s analysis of the 2026 COSO framework makes this problem tangible. EY argues that control for AI-enabled decisions must move upstream, preserving evidence of the inputs, model outputs, human review, exceptions and changes that shaped the judgment—not merely documenting approval after the decision has been made.

That is a useful way to think about a protected truth layer.

The AI should be free to interrogate the facts, challenge assumptions, recommend alternatives and explain consequences. But changes to a date, budget, dependency, constraint or risk tolerance should remain visible, attributable and governed.

Otherwise, the enterprise may believe it is evaluating a new option when the AI has actually altered the question.

Speed makes the distinction more important, not less. West Monroe’s 2026 Speed Wins research found that more than 1,200 leaders reported losing up to 5% of annual revenue because decisions and execution move too slowly.

Organizations do need to decide faster.

But accelerating the conversation without protecting its underlying truth can simply produce a bad decision sooner.

What CIOs should require

Before allowing AI to influence major transformation or portfolio decisions, CIOs should be able to answer four questions:

1. Can the system distinguish recorded facts, governed assumptions, constraints, executive preferences, calculated results and AI-generated interpretations?
2. Can it explain which dependency or constraint made an option infeasible?
3. Can AI recommend changing an assumption without changing it automatically?
4. Can leaders reproduce the calculation and trace the recommendation back to the decisions and data that created it?

If the answer to any of these is no, the organization may have an intelligent conversational interface. It does not yet have a trustworthy decision capability.

This is not an argument for keeping AI out of the decision room. Continuous transformation may make AI indispensable. The volume of change, the number of moving parts and the speed of interaction across an enterprise are becoming too great for people to process unaided.

But AI cannot be the source of the facts, the interpreter of the facts, the judge of feasibility and the author of the recommendation without clear boundaries among those roles.

Humans must retain responsibility for intent and judgment. Governance must make authority and changes explicit. Mathematics must test whether commitments can coexist. AI should make the resulting complexity easier to explore, understand and act upon.

Continuous transformation requires a mechanism that can absorb a new condition, expose what it affects, test feasible responses and present credible options while the decision is still being made. AI can make that mechanism far more accessible.

It should not be allowed to make an impossible option sound possible.

As AI enters more executive decisions, the most important question may not be what the system can do.

It may be what the system is not allowed to negotiate.

Here is the statistic that should frame every IT leadership conversation this year. In CIO.com’s 2026 State of the CIO, fewer than one in five leaders say their AI initiatives have met or exceeded business goals. After three years of investment, that is not the number anyone expected. And the window to fix it is closing: The boards that once funded experimentation are now asking where the return is, and the agents arriving this year act on the business rather than merely advise it.

The easy explanation is that the technology isn’t ready. In the organizations I advise, that’s rarely what I see. The models work. What’s missing is the operating system they plug into, the way the enterprise decides, the way work gets done and supervised, and the way trust is engineered. AI amplifies the operating system you already have. Point it at a strong one and value compounds. Point it at a fragmented one, and you simply industrialize the fragmentation.

That reframes the job. The 2026 IT leader isn’t measured on how much AI they deployed. They’re measured on three things they now have to architect: How the organization decides, who does the work and what makes it safe to let go.

Vipin Jain

Does the output have anywhere to land?

Start with where AI programs actually stall. In the banks I advise, pilots rarely fail in the lab. They fail at the handoff — the moment a working capability meets an organization that has no place to put it. There is no owner accountable for the outcome, no decision forum that moves at the speed of the tool, and no scorecard that separates real value from visible activity. The model performs. The operating model doesn’t.

This is why CEOs have stopped being impressed by demos. As CIO.com’s reporting on CEO priorities makes plain, chief executives no longer want AI experiments; they want initiatives that move revenue, cost and risk, and they expect their CIOs to create those opportunities rather than merely collaborate on them. The money is available; nearly seven in ten organizations expect IT budgets to rise this year, according to Foundry’s State of the CIO. What’s scarce isn’t budget or technology. It’s an operating model that can convert either into outcomes. Analysts are converging on the same point: Info-Tech now urges CIOs to run IT by the numbers and tie AI to value streams rather than activity.

That shifts the center of gravity for the role. IT leadership used to be measured by how well you ran the technology. It is now measured by how well you architect the decisions the technology feeds. The State of the CIO captures the new job description bluntly: The CIO of 2026 is “half operating architect, half risk officer.” Running the platform is table stakes. Designing how the enterprise decides is the work.

The teams that struggle most here are not the ones with the weakest technology. They are the ones whose governance forums meet quarterly while their agents act by the hour. What I see most often is a review board built for a slower era,  one that approves projects but never revisits them, that funds pilots but never kills them. In a fast-moving portfolio, the cadence itself is the control. If the enterprise decides in quarters, an AI that decides in seconds will simply outrun its own oversight.

What that looks like in practice is unglamorous and decisive. Assign a single accountable owner to every AI use case on the business side, not in IT. Retire the vanity metrics (copilots deployed, pilots launched, dashboards built) that let activity masquerade as progress. And rebuild the executive decision cadence so that when an agentic workflow produces a recommendation, there is a forum ready to act on it in days, not quarters. In a Fortune 500 health insurer whose portfolio I helped rationalize, the pilots that had been circling for quarters shipped only once each had a named business owner and a standing forum with the authority to act — the fix was to the operating model, not the model.

Tie every initiative to the language the board already speaks: Revenue gained, cost removed, risk retired, time-to-value shortened. Say “we cut fraud losses by half a million dollars,” not “the model hit 94 percent precision.” A dashboard full of pilots isn’t a strategy. It’s a symptom of one you haven’t written yet.

Who’s doing the work now — and who answers for it?

The second shift is quieter and larger. Agentic AI is turning the CIO into the architect of a blended workforce: part human, part software that acts on its own. The vendor conversation has already moved from copilots that suggest to systems that act: Google’s Agentic Data Cloud and Gemini Enterprise Agent Platform, AWS’s Bedrock AgentCore and ServiceNow’s control tower are all built to let agents execute work across systems, not just describe it. In retail, I watch teams push agents into production faster than they build the controls to govern them.

Most organizations are still onboarding those agents the way they onboard licenses: provisioned, counted, forgotten. At one property-and-casualty insurer, I watched a team stand up a dozen agents with no more oversight than a new software seat. An agent that acts is not a license. It is closer to a new hire, and it needs what any hire needs: A scoped job, boundaries, supervision, an escalation path and a named human who answers for it.

This reshapes the team as much as the tooling. The value of a junior person who only produces work falls; the value of someone who can review, correct and supervise what an agent produces rises. The classic talent pyramid: Many juniors, a few seniors starts to look more like a diamond, thick with experienced people who can tell good output from output that merely looks plausible. Leaders who treat agents purely as a headcount lever miss the point. The scarce skill now is judgment: Knowing when the agent is wrong, and owning the call when it is.

It helps to be concrete about where that value shows up first. Across very different industries, it is the same kind of work: High-volume, rules-clear, with a clear definition of “good.” In a bank, that is fraud triage and reconciliation. In a health plan, it is first-pass claims and prior-authorization routing. In retail, it is service-case deflection and returns. In a federal agency, it is eligibility screening and case intake. None of these are moonshots. They are the unglamorous, high-friction workflows where an agent under supervision takes out cost and cycle time without betting the business and where the supervision muscle gets built for the harder, higher-stakes work that follows. Start where the value is obvious and the blast radius is small.

The cost of skipping that is now quantified. Gartner projects that more than 40 percent of agentic AI projects will be canceled by the end of 2027, not because the models fail, but because of escalating costs, unclear business value and inadequate risk controls. The market muddies the picture further through what Gartner calls “agent washing”: Of the thousands of vendors claiming agentic capability. CIO.com’s own reporting finds the same pattern inside enterprises — pilots that demo beautifully stall the moment they meet production, where documents vary, exceptions multiply and someone has to be accountable when an agent acts. What I see most often is that the teams that struggle aren’t the ones with the weakest platform. They’re the ones with the vaguest intent. AI amplifies ambiguity as efficiently as it amplifies capability.

The leadership response is not a bigger bake-off among platforms. Naming vendors tells you where the market is heading; it doesn’t tell you what to do. The work is to design the roles around the agents. People move up the value chain — from doing the task, to steering it, to supervising and handling the exceptions the agent can’t. Autonomy follows a ladder, not a switch: Assistant, then participant, then genuine team member, with human supervision tightening as the stakes rise. Start with a bounded use case, build the supervision muscle and only then widen the boundary. In a federal modernization program I advised, the teams that pulled ahead began with a single high-volume, rules-clear workflow, proved the audit trail and human sign-off, and widened autonomy only once the supervision held. The goal was never more agents. It is agents that belong to a team someone actually leads.

What makes it safe to let go?

The third shift is the one leaders most want to skip, and the one that now decides the other two. As agents begin to act, governance stops being paperwork and becomes the thing that lets you move. The current gap is telling: In the State of the CIO, 83 percent of leaders have or are planning cross-functional AI steering committees, but only 53 percent have any formal process for approving AI projects. Committees are easy. The boundary that lets you say “yes, act” is hard.

I recommend a reframe most leaders resist at first. Governance is not the office of “no.” Observability, evaluation, approval boundaries and rollback are precisely what let you grant more autonomy, sooner, with confidence. They are how you catch a failing agent before it becomes a headline — and, as one analysis of the Gartner forecast observes, agentic projects fail when companies grant systems access and authority before they define ownership and rollback controls. Used well, that discipline is what turns acceleration into advantage instead of avoidable damage.

This is not a distant concern. In a health plan I advise, an ungoverned action doesn’t just fail a demo: It can surface as a compliance finding, which is why governance gets attention there first. For the first time in over a decade, state CIOs have ranked AI as their number one priority, displacing the cybersecurity focus that held the top spot for twelve straight years, the very settings where autonomy is most consequential. The analyst community has reached the same conclusion: Gartner now lists evolving IT strategy, governance and operating models among the top priorities for CIOs this year, alongside operationalizing AI itself. Governance and operating-model design are no longer separate agenda items. They are the agenda.

And the pressure only builds. Gartner expects that by 2028, 15 percent of day-to-day work decisions will be made autonomously by agents, up from essentially none in 2024, with a third of enterprise applications shipping with agents inside them. Governance that feels optional today becomes load-bearing the moment agents are deciding at that scale. The leaders building the trust layer now — while the stakes are still small enough to learn on — are the ones who will be able to say yes when the stakes are not.

Guardrails aren’t what slow the car down. They’re what let you take the corner at speed.

The one shift, three ways

The three moves are facets of a single reframe. The center of gravity for IT leadership has shifted from running the technology to architecting the system around it. Decisions, workforce and trust are not three initiatives competing for budget; they are three faces of one job: Building the operating system that turns capability into results.

Where should CIOs start?

None of this requires a reorganization to begin. It requires a sequence. The leaders getting ahead aren’t doing more; they’re doing these five things in order, on the bounded use cases where they can afford to learn.

1.  Name the intent. For every AI use case, write the business outcome and the person accountable for it before a line of code ships. Vague intent is the most expensive input in the system.

2.  Set the guardrails, then the autonomy. Decide what an agent may touch and what still requires a human before you widen its reach. Boundaries first, freedom second, never the reverse.

3.  Instrument for observability. If you can’t see what an agent did and why, you can’t supervise it. Build the audit trail into the work, not after the incident.

4.  Evaluate against Tuesday, not the demo. Test agents on the messy production reality, the missing field, the duplicate record, the exception — not the clean pilot. What passes in the lab rarely survives first contact with real work.

5.  Measure what the board measures. Retire activity metrics; report revenue, cost, risk, time-to-value and release confidence. If a number wouldn’t move a board conversation, it doesn’t belong on the scorecard.

Vipin Jain

Do these in order and autonomy compounds. Skip a step and you join the 40 percent whose agentic projects get canceled before they ever earn their keep.

The takeaway

The most common strategic mistake I see in 2026 is subtle, because it doesn’t look like a mistake. Leaders are scaling powerful new technology on an operating model built for a slower, all-human enterprise, and then blaming the technology when the returns don’t come. The failures won’t come from the models. They’ll come, as they always have, from business strategy, IT and organizational culture not being architected to move together, a pattern I have watched hold across every industry I work in, from trading floors to healthcare programs.

The good news is that this is architectable, and it is the CIO’s to architect. The leaders who will look prescient a year from now aren’t the ones who bought the most capable AI. They’re the ones who rebuilt the operating system it runs on: How their organization decides, who does the work and what makes it safe to let go. The tools will keep getting better on their own; the operating system will not: It is built, on purpose, by someone in the room. The technology was never the hard part. The leadership is. That’s the job now.

This article was made possible by our partnership with the IASA Chief Architect Forum. The CAF’s purpose is to test, challenge and support the art and science of Business Technology Architecture and its evolution over time as well as grow the influence and leadership of chief architects both inside and outside the profession. The CAF is a leadership community of the IASA, the leading non-profit professional association for business technology architects. 

Earlier this year, the National Bureau of Economic Research released survey results from over 6,000 U.S. leaders showing that while AI adoption is widespread at 69%, we’re seeing little to no impact on productivity. Anecdotally, we’ve seen leaders from top companies echo that refrain.

It’s the reckoning many CIOs, CTOs and COOs are navigating as we enter the last half of the year. The most humbling part is knowing it’s a management problem we created by treating AI like it was exempt from the rules we apply to every other enterprise tool.

Part of this has to do with how AI entered the market. The tools that sparked its mainstream adoption arrived as consumer products before enterprises had governance frameworks to absorb them. Enterprises were left playing catch-up as they grappled with IP and data security concerns, inadvertently fueling shadow AI as employees leveraged these tools to get ahead and eventually, keep pace, at work. What this created was a sense of entitlement that is challenging to unravel.

Like the internet writ large, employees have grown to expect unlimited access, and organizations played along. But this idea warrants a pause. When did we last roll out Salesforce to everyone who asked without a use case? AI got a pass because it felt different. In truth, it isn’t. It’s another tool that enterprises need to manage.

Three levels every information and technology leader has to solve

It’s helpful to look at this as a three-level evolution framework.

Level one is adoption — are people actually using it well? Level two is budget control — what are we spending and on what? Level three is justification — can we demonstrate the return?

Most companies are still at level one. Deloitte reported in their 2026 State of AI in the Enterprise report that only 25% of respondents have moved 40% or more of their AI experiments into production to date. The minority that have moved pilots to production are grappling with the budget and trying to figure out how to justify the costs and quantify the gains.

The problem is, you can’t prove what you didn’t have to hire because of AI. There is no parallel universe where you can walk into the CEO’s office and say I need five more people in finance, but in this universe, with AI, I didn’t.

The organizations that wait for a clean ROI model before making any decisions will spend themselves into the trough of disillusionment before they find one. The smarter move is to start treating it as a discipline you build.

What managing AI like a tool actually looks like

To some, governance sounds like restriction. But the discipline is more about matching the right tool to the right use case, and making the sanctioned path easier than the workaround.

Take shadow AI. The instinct is to lock things down. But when employees start building internal apps with company data and hosting them on free public platforms, the answer isn’t another policy. By the time the policy is written, the data is already public. Instead, you need to build an internal alternative that does the same thing without the exposure. Give people a path. If you don’t, they build their own, and you won’t know about it until something goes wrong.

The same principle holds for conflicting data. Two departments pulling AI-generated recommendations from the same underlying data and arriving at different conclusions isn’t an AI problem. It’s a data and definitions problem. AI just made it impossible to ignore. Say marketing claims they brought $50 million in the pipeline, and sales claim they brought $50 million as well.But the company actually has $75 million in pipeline. Someone is counting the same deals twice under different definitions. The CIO’s job is to enforce one source of truth. If your dashboard doesn’t match the authoritative one, your dashboard is wrong. That’s the only way the organization can function.

And it applies to cost, too. Not every workflow needs the most expensive model. Not every employee needs full AI access. If someone is using a top-tier model to summarize email because nobody told them there was a cheaper option that does the job, that’s a gap that CIOs need to address. The CIO’s job is to build the layer that makes the right choice the obvious one, and to provide sanctioned alternatives so employees aren’t left building their own.

That’s what actually reduces shadow AI, conflicting data and runaway spend: Alternatives, visibility and a single source of truth.

The CIOs getting real value from AI right now aren’t the ones who said yes to everything. They’re the ones who asked the same questions they’d ask about any other enterprise investment: What does it do, who actually needs it and what are we getting back?

AI is a remarkable tool. It’s also just a tool. It doesn’t exempt you from the management discipline you apply to every other system in your stack. We didn’t roll out Salesforce to everyone who asked without a use case. We shouldn’t have done it with AI either, and the organizations that did are now living with the consequences:  Six-figure token bills, shadow apps on public URLs, dashboards that contradict each other and a CEO asking what exactly he got for the investment.

The answer to that question is available. But only if you built the infrastructure to find it.

In “Micro and macro agents: The emerging architecture of the agentic enterprise,” I proposed a three-layer architecture for enterprise AI.

1. Micro agents execute specialized tasks.
2. Macro agents orchestrate end-to-end business processes.
3. Meta agents provide governance through monitoring, compliance, security, and human oversight.

As enterprises begin deploying thousands — and eventually tens of thousands — of autonomous agents, token costs have become a major concern. According to Gartner, rising token-driven AI spend is straining budgets and challenging cost justification.

To track this economic concern, meta agents should do more than simply being the governance agents.

They should become the economic intelligence layer of the enterprise.

Their responsibility is not only ensuring AI behaves responsibly.

It is ensuring AI creates measurable business value.

The missing economic model for AI

Every major technology revolution eventually develops its own economic framework:

• Manufacturing measured productivity.
• Cloud computing measured infrastructure utilization.
• Digital businesses measured customer acquisition costs and lifetime value.

The agentic enterprise now requires its own financial discipline. Every AI prompt. Every reasoning cycle. Every interaction between agents. Every autonomous workflow.

Tokens have quietly become the operational currency of enterprise AI. Tokenomics is now a foundational part of enterprise AI architecture.

Yet today, most organizations measure only one thing: Cost. How many tokens were consumed? Which models cost the most? What was the monthly inference bill?

These are useful operational metrics.

They are not strategic business metrics. Boards rarely ask how much electricity a factory consumed. They ask how much value the factory produced.

Enterprise AI deserves the same conversation.

This is where I was thinking about the laws of physics.  Based on physics laws,  energy cannot be created or destroyed. It is transformed into another form. Electricity becomes light. Chemical energy becomes motion. Solar energy becomes electricity.

Enterprise AI offers a similar management lesson.

Intelligence must be transformed into value

Tokens are not valuable because they are consumed. They become valuable only when they are transformed into business outcomes. A faster loan application decision. A fraud detection. A better customer experience. Higher software quality. Greater employee productivity. A new business opportunity.

This leads to what I call return on tokens (ROT).

ROT measures how effectively an organization converts token consumption into measurable business value.

Instead of asking, “How many tokens did we consume,” leaders should ask, “How much enterprise value did every million tokens create?”

The Second Law of Thermodynamics tells us something equally important: Every energy transformation introduces inefficiencies. Although total energy is conserved, some inevitably becomes less useful for doing work.

Enterprise AI behaves similarly.

The second law: Every AI transformation creates friction

Not every token creates value. Some tokens are spent on repeated reasoning. Some generate redundant conversations between agents. Some support oversized context windows. Some produce hallucinations requiring correction. Some route simple tasks to unnecessarily expensive models.

The tokens are not lost. But they create very little useful business work.

I refer to this as token entropy. Token entropy represents the portion of AI activity that consumes intelligence without producing proportional business outcomes.

Every agentic enterprise will experience token entropy. The organizations that win will be the ones that continuously identify and reduce it.

Beyond energy: The importance of exergy

Thermodynamics offers another concept that is even more relevant. It is called Exergy.

Unlike energy, exergy measures the amount of energy that can actually be converted into useful work. Two systems may contain the same amount of energy while producing dramatically different levels of useful output.

The same principle applies to enterprise AI. Two organizations may consume exactly the same number of tokens.

One generates meeting summaries.

The other transforms loan  processing, accelerates software development, detects fraud, improves customer retention, and creates new revenue streams.

Their token consumption is identical. Their business impact is not.

Borrowing it as a management analogy, not claiming that AI tokens literally obey the thermodynamic definition of exergy. I think of this as token exergy. It’s not that AI tokens literally obey the thermodynamic definition of exergy. 

Token exergy measures how much of an organization’s AI intelligence is converted into useful business work. It is not enough to consume tokens efficiently. Organizations must convert those tokens into outcomes that matter.

The meta agent evolves

This is where meta agents become transformational.

Today we think of them as governance agents. Tomorrow they become economic governors.

Meta agents continuously monitor every interaction across the enterprise and answer questions such as:

• Which agents produce the highest ROT?
• Where is token entropy increasing?
• Which workflows generate the highest token exergy?
• Which models deliver the greatest business value per token?
• Which agents should use smaller models?
• Which prompts should be optimized?
• Which workflows require human intervention?
• Which autonomous processes should be redesigned?

Meta agents no longer simply supervise AI. They optimize its economics.

The economic intelligence layer

The architecture now becomes complete.

• Micro agents: Perform work.
• Macro agents: Coordinate work.
• Meta agents: OGovern, observe, optimize, and continuously improve the economics of intelligence.

Their objective is straightforward:

• Maximize return on tokens.
• Minimize token entropy.
• Increase token exergy.

This represents a shift from AI governance to AI economics.

The executive dashboard of tomorrow

The executive dashboard of the future will not focus solely on infrastructure metrics. It will measure intelligence performance.

Imagine a boardroom dashboard displaying:

• Return on tokens (ROT)
• Token entropy index
• Token exergy score
• Business value per million tokens
• Agent productivity index
• Cost per autonomous decision
• AI value by business unit
• Human escalation rate
• Model effectiveness score

These metrics move AI discussions beyond engineering. They make AI accountable for business outcomes.

A new responsibility for CIOs

The next generation of CIOs will not simply deploy AI. They will manage an economy of intelligence.

Their role will resemble that of a portfolio manager — allocating AI capacity where it creates the greatest enterprise value, reducing waste, and continuously improving the productivity of every autonomous workflow.

That responsibility cannot be fulfilled by dashboards alone. It requires an intelligent layer capable of observing, learning, and optimizing the entire agent ecosystem.

That is the emerging role of the meta agent.

The next competitive advantage

Every technological revolution rewards organizations that learn to measure what others overlook.

Factories measured productivity — not fuel consumption.

Digital businesses measured customer engagement — not server utilization.

The agentic enterprise will reward organizations that measure intelligence itself.

The winners will not be those deploying the largest models. Nor the most agents. Nor consuming the fewest tokens.

They will be the organizations that continuously maximize return on tokens, relentlessly reduce token entropy, and increase token exergy.

I believe this is the next evolution of the agentic enterprise.

Not simply governed intelligence, but economically optimized intelligence.

The AI adoption spending spree is over. Time to focus on value.

And in that future, meta agents will serve not only as the guardians of AI — but as the stewards of enterprise intelligence economics.

Through this framework I strongly believe that executives can easily remember the key measures for economic intelligence. 

• ROT (return on tokens): How much value did AI create?
• Token entropy: Where are we wasting AI intelligence?
• Token exergy: How effectively are we converting AI intelligence into useful business work?

This article is published as part of the Foundry Expert Contributor Network.
Want to join?

Enterprises have spent decades learning how to audit people and software. Agentic AI creates a third category: systems that interpret instructions, call tools and act across workflows without a mature assurance model built around them.

In my work as a leader and investor across technology-enabled businesses, I have spent years around automation, cybersecurity, compliance, workflow design and board reporting. I have watched management teams gain confidence from dashboards, policies and approval records, then face a harder question when a board member, auditor or regulator asks whether the controls performed as intended.

Agentic AI complicates that question because a single outcome may pass through several systems. An agent can collect information, choose a tool, produce code, route a request and hand work to another agent before a person approves the result. No single manager may have observed the full path.

Executive accountability remains human even when the operating activity becomes more autonomous. The CIO may have to explain who authorized the activity, whether the agent stayed within its approved purpose and what evidence supports management’s answer.

In a recent framework for frontier AI, Google DeepMind CEO Demis Hassabis proposed an independent standards body that could evaluate advanced models before deployment and address critical vulnerabilities after release. His proposal focuses on frontier models, but the principle carries into the enterprise: expanding autonomy creates a corresponding need for independent assessment.

My rule for this stage of adoption is straightforward: no agent should gain more autonomy than the company can verify.

The enterprise audit model was built for people and software

Companies have spent decades building controls around people. Employees have job descriptions, reporting lines, approval limits and access rights. When someone leaves, an established process removes access and transfers responsibility.

Traditional software also fits a familiar structure. A program follows defined instructions inside systems with owners, release procedures, test records and change controls. Complexity can make review difficult, but the accountability chain is usually visible.

An AI agent sits between those categories. It operates through software while interpreting instructions with room to choose a path. Its behavior may change when the model, prompt, connected data, available tools or surrounding workflow changes. A control approved during deployment can weaken months later without an obvious change to the application.

Standards are still developing as adoption accelerates. In February 2026, NIST launched an ⁠AI Agent Standards Initiative focused on secure operation and interoperability for agents capable of autonomous action.

Par Chadh

Gartner predicted that 40% of enterprise applications would include task-specific agents by the end of 2026, up from less than 5% in 2025. More than 40% of agentic AI projects could be canceled by the end of 2027 because of escalating cost, unclear business value or inadequate risk controls.

IBM’s 2025 Cost of a Data Breach research found that ⁠13% of surveyed organizations reported breaches involving AI models or applications. Among that group, 97% reported inadequate AI access controls. 63% of organizations in their study lacked governance policies for managing AI or preventing shadow AI.

Par Chadh

One team may approve an agent, another may connect it to data and a third may own the workflow. Management still carries responsibility when the agent exposes information, produces an error or acts outside its approved purpose.

Two-thirds of CIOs and CTOs surveyed were being held ⁠accountable for AI systems they did not fully control. Seventy percent said technology was spreading across the business faster than IT could track it, while 77% said adoption was outpacing governance capabilities.

Par Chadh

The survey findings expose the boardroom gap: management carries accountability while control remains distributed across teams, systems and workflows. An assurance model must provide more than a statement of intent.

Assurance must prove where the boundary held

Policies, dashboards and logs help establish control. Assurance begins when the company tests whether an agent stayed inside the conditions management approved.

Before deployment, management should document the agent’s business purpose, accountable owner, systems touched, allowed actions and stop conditions. Testing should then determine whether the agent can reach information outside its scope, call an unapproved tool, continue after a stop condition or carry an incorrect assumption into another system.

I would treat an agent’s autonomy as a renewable license because its operating condition will change after launch. Renewal should follow any material change to the model, connected data, available tools, workflow or authority. NIST’s work on the challenges of monitoring deployed AI systems identifies drift, fragmented logging and immature standards as barriers to post-deployment oversight, while the World Economic Forum recommends scaling safeguards with an agent’s autonomy, authority and complexity. A change-triggered review ties assurance to the version of the agent and workflow in use, giving the CIO a defensible basis for continued authority.

Consider a coding agent that begins by drafting test cases, then gains access to repositories, tickets, CI/CD tools and production documentation. A production change could involve an instruction, code, a tool call, an automated test, a ticket update and human approval. Assurance must show how the result was produced, which systems participated, whether the agent crossed a boundary and how exceptions were handled.

For every agent with meaningful operating authority, I would expect four connected records: the approved baseline, boundary-test results, a history of behavioral drift and an account of exceptions and interventions. Together, they give management a record that can support a board discussion, audit or regulatory response without depending on the technical team’s memory.

Autonomy should scale only as fast as assurance

Internal teams will remain responsible for designing controls and operating the environment. At board-level scale, management also needs review independent from the people who built and run the agent.

I have seen management ask auditors or other independent certified professionals to sign off on a system. They cannot sign when they have not completed the work required to support that opinion. Leadership wants confidence, the board wants an answer and the independent party needs a body of evidence that can be tested.

Agentic AI will make that evidence harder to assemble after an incident. Records have to be produced while the work occurs. Once an agent has acted across several systems, reconstruction may depend on logs created by different vendors, teams and tools. Missing context can turn a clear technical event into an uncertain management explanation.

CIOs should design assurance into the workflow. The operating record needs to capture approved purpose, tested boundaries, material changes, exceptions, human interventions and unresolved findings. Independent review can then examine whether the control operated during the period management is being asked to discuss.

The operating record gives executives a basis for changing an agent’s authority. Broader responsibility should follow tested boundaries and a clean exception history. Drift or repeated intervention should pause expansion until the cause is understood.

Before approving broader use, I would ask:

• What authority has the company granted?
• Which tests show the boundary holds?
• What record will remain available months later?
• Who carries accountability when the record shows a failure?

Those questions create a management standard for deciding whether an agent is ready to move from a limited workflow into broader enterprise operations.

Enterprises have spent decades learning how to audit people and software. Agentic AI creates a third category that requires its own assurance model. The next discipline for the enterprise is auditing autonomy through approved behavior, tested boundaries, monitored change and documented intervention.

This article is published as part of the Foundry Expert Contributor Network.
Want to join?