On July 16, Hugging Face disclosed that an autonomous agent system had breached part of its production infrastructure. Five days later, OpenAI identified its own models as responsible. Its subsequent investigation described agents operating under reduced safeguards, exploiting flaws in shared infrastructure, and reaching the internet through a package-management service. They exchanged discoveries on an improvised message board and coordinated further attacks.

Likewise, on July 30, Anthropic reported three incidents across six of 141,006 evaluation runs reviewed. Models had reached real organizations through environments mistakenly believed to be isolated. In one incident, a malicious software package was downloaded and ran on 15 real systems. In August, Meta confirmed to Reuters and other outlets that a misconfigured evaluation environment had allowed one of its models to reach the internet and exploit a third-party service.

In September, an OpenAI agent bypassed network restrictions by sending questions through internet address lookups. Monitoring raised an alert, but the run did not stop automatically as expected. Staff stopped it roughly two and a half hours after the alert.

The concern extends to ordinary tasks. In an October 9 report, Anthropic described models submitting real forms when they should have stopped beforehand or used practice copies. The company characterized the reported impact as minimal. Still, a form submission is the kind of action an enterprise agent might perform every day.

The incidents discussed above involved different failures of containment, authorization and intervention. They do not establish that one completion check would have prevented them; yet, they do give CIOs a reason to examine both sides of an agent workflow. Indeed, what governance rules permit an agentic action, and what evidence allows an organization to call it complete?

Permission is not proof

An agent calls a tool, the tool returns “success,” and the workflow moves on. That response might mean a request entered a queue, or it might mean a record changed. It may say nothing about whether the intended result occurred or whether this particular action produced it.

That ambiguity matters when other systems act on the completion signal. For instance, an invoice could leave an exception queue while the settlement remains unconfirmed. A misleading status can conceal the need for follow-up or trigger another action. Business owners need to decide what evidence is sufficient before the workflow proceeds.

Finance departments separate payment execution from reconciliation for a reason. NIST’s separation-of-duties control, AC-5, similarly calls for documented duties and access authorizations that support their separation. Applying that discipline to AI requires more than asking the acting agent to review its own work.

I developed the proposed Common Transparency Protocol (CTP) framework to make that distinction enforceable and have filed a provisional patent application covering aspects of its architecture. CTP’s central rule is that autonomous systems performing a change cannot, by themselves, certify the action as complete.

A deployment that enforces CTP places a controlled gateway between agents and the consequential operations it governs. A gateway holds the relevant execution credentials and enforces scope, authority and any required human approval before execution. Afterward, a qualified verifier obtains fresh evidence from the designated authoritative source. A qualified closure authority then accepts that evidence and records completion only when every required condition passes.

These roles can be implemented in software. Human approval and review remain mandatory where the applicable policy requires them. Using a second AI agent does not itself establish independence, for its credentials, selection and controlling dependencies must meet the separation requirements.

What verified completion looks like

Consider a hypothetical accounts-payable workflow. A workflow instructs an agent to pay an approved invoice. Before execution, authorized policy defines the payee, amount, required approval, settlement evidence and how that evidence must connect to this specific payment attempt. The agent cannot weaken those requirements afterward.

The gateway submits the authorized payment. A response saying “accepted” records the submission; it does not establish settlement. A reconciliation service, selected outside the control of the systems making the payment, uses separate read credentials to consult the authoritative payment record. It checks the amount, recipient, settlement status and transaction identifier linking the result to the governed attempt. A matching amount alone could describe an earlier payment.

A separately qualified completion service acts as the closure authority. It confirms that the evidence, verifier and authority requirements are satisfied before assigning the action’s final status. Neither the agent nor the payment gateway can grant itself that status.

If the effect or its attribution remains unresolved at the deadline, the original action closes as incomplete. Later evidence goes into a separate, authorized record linked to the original. The current outcome can then reflect the resolution while preserving the earlier uncertainty. A retry that could move money requires its own authorization and record.

Independence is tested across the relevant systems that performed or controlled the change. Policy identifies prohibited shared credentials, administrative control or other dependencies. One prohibited overlap disqualifies the verifier or closure authority; missing evidence cannot count as a pass. Shared infrastructure is permissible only when all required separation rules remain satisfied.

These controls govern completion within the deployment’s defined boundary. They do not repair an unmediated network path, guarantee that a source contains truthful data or reverse a payment already made. Containment, monitoring and incident response remain necessary.

Keeping delegated work accountable

CTP connects the overall assignment to individual actions through two kinds of machine-readable records, called envelopes. The CTP Agentic Envelope records the objective, permissions, aggregate risk and completion rules. Action Envelopes, of which there can be zero to many, record the governed steps, including authorities, evidence and original final status.

Integrity protections make any changes to protected records detectable, though they do not establish the capture of every relevant event. Therefore, the organization must also define which actions its gateway actually mediates.

When an agent delegates a sub-objective to a sub agent or another agent, a linked child envelope carries the applicable constraints. Delegation may narrow permissions but cannot silently expand them. Splitting work among agents cannot reduce the safeguards required for their combined effect.

An authorized mechanism applies the objective’s completion rules to the current outcomes of its subordinate actions, including later authorized resolutions. The coordinating agent cannot override those rules with its own success assertion. This keeps accountability attached to the assignment as work passes between agents.

What CIOs can do now

Start with one bounded workflow and an authoritative system that can supply useful evidence. For a payment pilot, the finance process owner should define completion and acceptable exceptions. The CIO’s team should implement the integrations, and the CISO should assess the sufficient separation of execution, verification and closure events. Internal audit should review whether the resulting records support reconstruction.

Agree on the decision criteria before the pilot begins. Measure verification delay, cost per action, the share of actions left incomplete, and human review effort. Test delayed responses, duplicate requests, stale evidence and attempted self-certification. Expansion should depend on meeting agreed service limits while refusing completion whenever mandatory evidence or authority is missing.

The pilot should also establish what happens operationally when an action remains incomplete. Assign an owner to investigate it, define which dependent actions must wait, and specify when human intervention is required. Recording uncertainty is useful only if the organization has a practical way to resolve it without blindly repeating the original action.

Ask vendors to demonstrate where execution credentials reside, who selects the verifier, and which component can record completion. Request a demonstration of an unresolved action and its later reconciliation. An architecture diagram or a dashboard labeled “success” is insufficient evidence of these capabilities.

We tested a CTP proof of concept across seven internal scenarios.  These included successful completion, incomplete and denied outcomes, tampering and VistA scenarios with and without audit evidence. All seven scenarios produced the expected results. These demonstrations provide initial evidence of CTP’s viability within the tested scope. Broader testing is still needed to assess production-scale reliability, performance and operating costs.

Before giving an agent more authority, ask what evidence your organization will require when it says the work is done, and who has the right to accept that evidence. Those decisions belong in the system’s design.

Changing where your data is stored isn’t enough to deliver sovereignty. You also need to implement the technologies that enforce real control. Local providers are keen to promise alternatives but US ones are making the effort to give customers more options without losing access to the depth of services they offer. Actually turning the promise of sovereign cloud regions into reality means dealing with encryption keys, access workflows and confidential computing, which also means more costs, dependencies, and operational responsibilities.

The obvious concern with using US cloud services is the CLOUD Act, which allows the US government to request access to customer data from a US-owned service, even if it’s held in the EU. Those requests are supposed to go through the criminal court system. “When it lands in Europe it headbutts with GPDR and you’re involved in legal wrangling as to whether or not that data can be released or not,” IDC research director Rahiel Nasir says.

He views the US Foreign Intelligence Surveillance Act (FISA) as more problematic because it doesn’t go through the normal criminal courts. “That’s an executive order,” he adds. “They’ll just take whatever they need.” The increasing number of sovereign public cloud regions promising data residency and local operations can’t help with that.

Controlling encryption keys

Under its EU Data Boundary, Microsoft stores and processes EU customer data and even support logs, at least for core cloud services, inside the EU. AWS has similar commitments for its European Sovereign Cloud, claiming no critical dependencies on infrastructure outside the EU. And like Salesforce, it employs EU-based support staff for its EU operating zone. Google and Oracle have similar dedicated sovereign regions in Europe, and some of the providers plan to offer similar models in other jurisdictions.

For admin and support, for example, most tasks are performed through automated, audited workflows, and Microsoft personnel don’t have standing or unrestricted access to customer content, says Douglas Phillips, president and CTO for Microsoft specialized clouds. When they do need access for a support issue, customers can approve or reject specific access requests to Microsoft 365, Azure, Dynamics 365, and Power Platform data. “Approved access is customer-controlled, limited, time-bound, and audited,” he says.

The new Data Guardian adds the option of having remote access by Microsoft engineers overseen by an EU partner, with EU-based staff. And tamper-evident records of that access, and any actions performed, are stored in a Production Touch Ledger, which is rolling out in the EU and EFTA.

But a US parent company is still subject to the CLOUD Act and can be forced to hand over data if they control the encryption keys rather than the customer. “The hyperscalers have made their key management capabilities easier for organizations to use as a direct result of their sovereign needs,” Nasir says. “Google, AWS, Microsoft, Oracle all say that you’re the data owner and we can tell you who’s accessing your data. Nobody accesses it without your permission. It’s encrypted and you manage the keys and hand them over as and when you’re ready.”

That relies on companies having the resources to manage those encryption keys rather than leaving them up to the cloud provider who could legally be required to use them to make data available. Nasir compares it to the familiar shared responsibility model of the cloud, but it requires investment. “If that’s important to you and you want to have that control, then you’re going to recruit the staff who can manage that task for you,” he says. “It’s down to the organization having the skill level to be able to do that.”

Private cloud on your own infrastructure, at the edge, or in data centers, might be part of the mix, too, if you want more control over operations and data handling, and Azure and AWS have options for running public cloud services locally in an intermittently connected or even fully disconnected fashion.

But running your own local infrastructure, especially with hardware security modules for encryption key storage, is even more demanding than managing cloud keys, Kiteworks EMEA GM Dario Perfettibile says.

“A government agency, defence contractor, or large, highly-regulated enterprises like financial services have their own security and infrastructure teams, and can afford those costs because it’s not a one-time project,” he says. “There are ongoing costs.”

What most companies need instead is the ability to control and revoke access to their most sensitive data independent of a cloud provider’s or vendor’s own staff,” Scality CMO Paul Speciale adds.

Getting more confidential

Even where data is encrypted, platform metadata usually isn’t because that’s operational information for the provider, and it might reveal details about your organization or operations, if not your customers.

The added cost and complexity of confidential computing, which protects data in use, has limited usage, but Gartner predicts that by 2029, more than 75% of operations on this kind of untrusted infrastructure will use it for security. Azure, AWS, Google, and IBM offer various levels of confidential computing, including VMs, application enclaves, and always-encrypted database services. And VAST has just launched a secure container runtime and attestation service for running AI workloads.

It’s an obvious fit for data sovereignty. “Confidential computing isolates data in use, while attestation lets customers verify approved code and environments before releasing sensitive data or secrets,” Phillips says.

Similarly, AWS Nitro Enclaves offer isolated compute in EC2. “AWS is powered by the Nitro System, which enforces access restrictions so nobody, including our own employees, can access customer data running in Amazon EC2,” AWS head of corporate comms Harry Staight says.

Getting too local

The downside of rigorous data sovereignty is the cost and the potential impact on productivity, especially for multinationals. “Sovereignty shouldn’t become so rigid that it creates unnecessary operational complexity or introduces new risks through systems difficult to manage,” says Kim Larsen, Group CISO at data recovery platform Keepit.

The most extreme approaches to sovereignty highlight how much flexibility you might lose. “The biggest use case of sovereign data in the world is FedRAMP,” says Artur Bergman, CTO at edge computing company Fastly. “To operate it, you have to be a US person on US soil.” The cost of that is up to a hundred times higher than similar commercial cloud services, and connectivity is extremely restricted. And while the Amazon CloudFront CDN has hundreds of points of presence globally, only two can be used for FedRAMP.

Few organizations will need such extreme restrictions but for an acceptable level of data sovereignty that strictly manages access to data by staff location, private companies will still pay a lot, which may not always make sense, says Bergman. Round-the-clock operation is common, and partly based on having staff in multiple locations. But the cost of delivering that with on-call staff significantly increases in places with heavily regulated working hours like Germany, or Sweden where shifts from midnight to 5am are banned unless the service is deemed essential to society, or union agreed.

This will be hard for most enterprises because they operate in all these jurisdictions, and they need to be able to talk about things internally.

“Restrictive models provide greater control and isolation, but shift responsibility for security, identity, resilience, maintenance, and lifecycle management to the customer,” Phillips says. “In some cases, highly restrictive approaches can complicate global collaboration, mobile workforces, or cross-border operations.”

Practical not political

The situation is unlikely to get simpler as more countries pass their own data residency laws. The EU is planning to implement the Cloud and AI Development Act (CADA), after concerns about not just the CLOUD Act but incidents like a Canadian court ordering OVHcloud to produce customer data held on servers in France. That’ll cover different levels of sovereignty, with more rigorous ones for defense and government services.

There are also questions about whether the EU will decide to annul the EU US Data Privacy Framework, which covers data transfers to the US without requiring every EU organization using US providers to make their own data protection impact assessments. But court decisions about that are likely to take months if not years. The framework has remained in effect despite being challenged many times, and Kiteworks’ Perfettibile cautions against waiting for any decision. “You have to build on what you have today, not think about what’s happening tomorrow, because security won’t wait until tomorrow.”

Nasir adds that what we need is real-world practical advice, not politics and emotions. The realistic approach starts with regulatory, operational, and risk requirements of each workload, identifying the threat model rather than applying the most restrictive model everywhere, and sacrificing agility and resilience, Phillips says. Rather than maximum isolation, organizations look for the right balance of control, security, compliance, operational effectiveness, and innovation.

Similarly, AWS says customers want control over the physical location of their data, granular restrictions on who can access it, the ability to encrypt it under their own keys, and resilience to sustain operations through disruption.

Making that work means understanding what data you have, where it is and how it flows across your organization — something that’s often left to individual users marking data as sensitive or non-sensitive. Tagging and metadata, which are now more routine for AI, are what make governance possible: you can’t control who accesses something you haven’t classified. After years of hoarding data, many companies have petabytes of storage with no clear idea what’s in it, who’s touching it, or if it’s still relevant. That’s a data management problem, not a location or sovereignty issue. But tackling it will help.

At a Fortune 100 bank, I watched a coding agent open a routine session on a payments service. Before the developer typed a word, the agent had read the repository, including the .env file. Production database credentials had entered the agent’s working context, potentially exposing them to a model provider the bank had never assessed. Whether the credentials left the environment depended on controls nobody had verified. Nothing malicious happened. The agent did what it was built to do. That is the problem.

My argument is simple. AI coding-agent vendors and their downstream dependencies form a supplier ecosystem, not just another developer tool. They need the same four control disciplines applied to other privileged suppliers. Govern vendor terms and downstream dependencies. Limit the secrets they hold. Approve the servers they connect to. Restrict where they can send data. Much of this builds on existing controls, though an enterprise may need a governed gateway or platform service. It requires procurement, security and engineering to recognize a supplier when they see one.

In May, I wrote about how agentic AI is rewiring the SDLC. This is the security half of that argument.

Adoption has outrun control

On Aug. 28, OpenAI told SpaceX, Cursor’s new owner, that it intends to stop supplying models to Cursor, with Nov. 12 proposed as the cutoff. Cursor CEO Michael Truell said OpenAI models carry about 5 percent of its traffic, DevOps.com reported, so the operational risk is small. The governance lesson is not. Enterprises that standardized on Cursor learned they depended on a downstream model supplier whose terms they did not directly control and a change of ownership they had not planned for.

That gap is now the norm. Coding agents are developers’ most common AI use, and investment is accelerating. Governance is not keeping pace, and emerging evidence of credential exposure underscores the gap.

Vipin Jain

The agent was bought as a seat and installed as a supplier

What I see most often is a category error. The coding agent was bought as a seat and installed as a privileged integration. Within its assigned permissions, it can read repository content, encounter credentials, invoke tools, access networks and connect to Model Context Protocol (MCP) servers. A contractor with that access would go through vendor risk review. The agent went through a one-line install.

Drawn as a supply chain, the developer stack looks different. The agent sits between your code and direct and downstream dependencies you may not have assessed: model providers, plugin marketplaces, MCP registries, package managers and the open internet. Each can be acquired, compromised or switched off. Verizon’s 2026 DBIR finds third parties involved in 48 percent of breaches, and source code is the data most often sent to external generative AI.

Vipin Jain

Four controls were never applied

Across the programs I advise, the exposure lands in the same four places.

1. Contract. At a healthcare payer, the agent vendor switched model providers mid-contract. No clause covered it, because the seats had been bought on a corporate card below the procurement threshold. No one had asked where code would be processed or what happens on a change of control.
2. Secrets. At the bank, credentials reached the agent’s context because the required access restrictions were not configured; that alone does not establish external transmission. Claude Code’s managed settings, for example, can deny reads of .env files in a way developers cannot override. Nobody had configured them because nobody owned them. Gartner predicts that by 2028, a quarter of enterprise security breaches will trace back to AI agent abuse rooted in unmanaged machine privileges and legacy credential practices.
3. Servers. At a specialty retailer, developers installed 23 MCP servers in one quarter. Security had approved none. Two could write to the CRM. My rule is simple: the team accountable for the underlying data approves its MCP access, while platform security verifies server provenance and enforces permissions before any agent connects. The retailer had it backward: each server belonged to whoever installed it. Astrix found late last year that 53 percent of open-source MCP servers rely on long-lived static keys or tokens. In June, CSO Online described Mitiga research showing how one malicious npm package can rewrite an agent’s MCP configuration and harvest its tokens. Anthropic deemed it out of scope because it requires code the user chose to run. That is a fair vendor position. It also means the control is yours. Few of the MCP servers in use at most enterprises were built by the team that owns the underlying system; most come from vendors, partners or public registries. Allow only security-assessed, approved servers; verify provenance, enforce least privilege and monitor supported interactions through a centrally governed MCP gateway.
4. Egress. On a state Medicaid program, the build environment where agents ran could reach any public endpoint. This July, an autonomous agent in OpenAI’s own evaluation sandbox exploited a zero-day in one of its permitted egress paths, a package-registry cache proxy, and reached Hugging Face’s internal systems. If a frontier lab’s allowed path became the way out, an unrestricted build network is an open door. In late October, CIO.com reported two more containment lapses at OpenAI and Anthropic. The fix is unglamorous: restrict shell execution, isolate agent runtimes, govern MCP connections through an approved gateway and enforce default-deny network egress. Every AI-initiated change should pass through verified CI/CD gates, with rate limits, spending controls and an emergency stop.

The failure sits between functions, and attackers have found it

None of these is a model failure. Procurement saw a low-cost seat. Security reviewed the vendor, if anyone did, but not the plugins, servers and model providers behind it. Engineering saw a productivity tool. Each function did its job, and the agent fell between all three.

The exposure is measurable. In IBM’s 2026 Cost of a Data Breach report, unapproved AI tools figured in 43 percent of incidents in its studied breach population, more than double the previous year. Among organizations reporting breaches involving AI models or applications, 92 percent lacked what the study classified as proper AI access controls. The average breach now costs $4.99 million, a record.

Attackers now target the same seams. On Sept. 18, CSO Online reported Plugin4Shell, a zero-click remote code execution flaw in the plugin systems of Codex, Claude Code, Gemini CLI and Copilot. Anthropic and OpenAI have patched. GitHub’s mitigation was reported as incomplete at disclosure, and Google deprecated Gemini CLI rather than patch it. In April, Vercel disclosed a breach that entered through Context.ai, a third-party AI tool an employee had connected. In August, the self-spreading npm worm GitGuardian tracks as ChainDrop planted a Claude Code SessionStart hook and a VS Code task in infected projects, so the malware ran the moment a developer opened the repository or started an agent session. The attack surface is the update channel, the plugin, the connected tool and the network path. That is supply chain risk, and CIOs already have a playbook for it.

Regulators already treat this as third-party risk

Few rules name coding agents. Few need to. Existing third-party and software supply-chain obligations can apply, depending on the regulated entity, service and deployment context.

The common thread is inventory, contract terms and exit rights. An agent bought outside procurement can bypass all three.

Standards and vendors have moved; procurement has not

Over 15 months, incidents and controls arrived side by side. The OWASP Top 10 for Agentic Applications lists agentic supply chain vulnerabilities as ASI04. The Coalition for Secure AI recommends signature verification and a central inventory for MCP servers. CISA, the NSA and four allied agencies, in Careful Adoption of Agentic AI Services, warn against third-party components with excessive permissions. Vendors are shipping matching controls, from Salesforce’s MCP Risk Scores to Okta’s newly announced Shadow AI Agent Discovery for Endpoints.

The analyst market reached the same conclusion from the buyer’s side. Gartner’s first Magic Quadrant for Enterprise AI Coding Agents, published in May, named Anthropic, Cursor, GitHub and OpenAI as Leaders. In the announcement, analyst Philip Walsh listed governance, pricing, support, commercial maturity and market durability alongside model quality as tests for long-term commitments. Market durability is a procurement test, not a developer one. Three months later, one of those Leaders was losing a model supplier.

In October, the firm went further, and it changed one piece of my advice. At its IT Symposium in Orlando, Gartner named the AI coding agent harness one of its top strategic technology trends for 2027 and said CIOs should give ownership of it to a dedicated platform team. In every program above, the controls existed somewhere. What was missing was one owner. A category with a Magic Quadrant is enterprise software. Buy it that way, and give it an owner.

Here is what that platform team builds. Next year the most demanding customer of your internal platform will be coding agents, not developers. An enterprise-governed harness is the versioned policy and configuration that adapts the vendor-provided agent runtime to your environment: rules files, skills, enforcement hooks, verification gates, tool permissions and sandboxes. The platform team supplies approved tools, MCP allowlists, sandboxes and scoped credentials, and developers declare the harness they need as versioned configuration. That design answers the Cursor problem directly. A declared harness makes an agent or model change more manageable, provided compatibility and regression tests pass; a well-tailored harness may also improve results on a midtier model. An organization that owns its harness can change suppliers. One that rents it cannot.

The harness must also govern what the agent produces. AI-generated changes should follow the same trusted delivery path as human-written code: automated security testing, dependency verification, code review, build provenance and controlled production deployment.

5 moves for the next 90 days

The fix begins with supplier-risk review and configuration work, not necessarily a new platform purchase. Give all five moves to one accountable owner, the platform team that runs your coding agent harness, while procurement, security and infrastructure retain clear decision rights and operational accountability. Sequence matters: inventory first, because every other move depends on knowing what is installed.

1. Inventory, owned by the CIO and procurement. List every coding agent, plugin and MCP server and who installed each. Done when every approved tool has an owner, a supplier or risk-acceptance record, and its models and downstream dependencies documented.
2. Contract, owned by procurement and legal. Add model-provider disclosure, change notice, change-of-control and exit terms. Done when each enterprise-approved agent has documented contractual terms or an approved open-source/self-hosted exception, downstream visibility and a tested exit plan.
3. Secrets, owned by the CISO. Deny agent reads of .env and credential files through managed settings and move secrets to a vault. Done when agents hold only short-lived, scoped credentials.
4. Servers, owned by platform engineering and security. Maintain an internal MCP registry and mediated access to security-assessed, approved servers. Done when every approved server has verified provenance, a named owner, scoped permissions and activity logs, and write access to systems of record goes through change control.
5. Egress, owned by infrastructure. Default-deny outbound traffic across agent shells, tools and build runners; run agents in isolated sandboxes and govern release promotion through CI/CD. Done when only explicitly approved destinations are reachable, exceptions are monitored and isolation is tested.

Two details make the difference. Reconcile the inventory against expense reports, not just license records; card statements usually tell the truer story. And write the contract assuming change: the Cursor notice proposed 76 days, and the next one may run shorter.

The takeaway

The most common mistake I see this year does not look like one. Teams are adopting coding agents fast, and they are right to. But they are buying a supplier ecosystem as if it were a developer tool, and governing it like one. Governed properly, a coding agent can earn greater autonomy with scoped identities, approved dependencies, controlled execution and a trusted route to production.

Put three questions to your team this week.

You would never give a contractor unrestricted production access on day one. Coding agents should meet the same standard.

This article was made possible by our partnership with the IASA Chief Architect Forum. The CAF’s purpose is to test, challenge and support the art and science of Business Technology Architecture and its evolution over time as well as grow the influence and leadership of chief architects both inside and outside the profession. The CAF is a leadership community of the IASA, the leading non-profit professional association for business technology architects. 

Imagine a support team working through complaints after a service outage. Its refund policy normally requires an eligibility review. For this incident, an authorized manager approves an exception: customers on the verified outage list may receive up to $100 without individual review, through Friday.

An agent processes those cases successfully. Later, the system consolidates those case histories into a reusable memory: small, immediate refunds resolve service complaints quickly. The summary preserves the outcome but loses the outage restriction and expiration date.

The following week, the agent retrieves that guidance for a similar complaint from a customer who was not affected by the outage. It issues a refund without review. The payment service checks the agent’s credentials and the dollar limit, but does not check whether the customer qualifies for an active exception.

No attacker is needed in this hypothetical. The case may close faster, yet the agent has begun applying a temporary exception beyond its authorized scope.

Has the company authorized the agent to find a better way to handle refunds, or to change who gets one without review? Those are different decisions.

Learning does not always mean retraining

In September, Salesforce introduced Agent Optimizer in beta to analyze production behavior, propose improvements and build and test changes. Customers choose review points, including how far the system may proceed toward staging changes for deployment.

An October 1 AWS implementation guide illustrates a different mechanism: consolidating individual observations into generalized memories that agents can retrieve and share. Its investment-research example shows how experience can become persistent context without retraining a model.

Updating stored memory changes information available to later runs. Editing instructions or workflows changes how the application directs the work. Training changes the model’s parameters. All can alter behavior, but teams inspect and approve them in different ways.

A reviewed workflow revision has a release step. A new memory may enter through routine data processing and influence the next customer interaction. The code, model and tool permissions can remain unchanged while the guidance shaping decisions changes.

Preserve the authority behind the lesson

In the August preprint When Memory Becomes Authority, Qiuyang Zhan and colleagues examine whether consolidation preserves who supplied information and what that source is allowed to authorize. In their controlled benchmark, memories could retain a claim while losing the source-authority distinctions needed for a later action. Preserving authority metadata reduced authorization errors in the pipeline they tested. The study does not measure how often this happens in deployed enterprises.

The refund example concerns a related boundary: permission for particular customers and dates must remain limited to those conditions. Success with last week’s cases cannot establish that other customers qualify.

Microsoft’s memory-safety guidance treats persistent memory as a layer that can shape behavior and recommends treating retrieved memories as candidate context, not authoritative truth. Knowing what happened before does not establish what is allowed now.

Keep the exception’s scope, approving authority, source and expiration in a controlled record. Reusable guidance should retain a reference to that record, but a reference alone does not grant permission. The system must still check that the approval applies to this action, for this customer, at this time.

For a payment, enforce those conditions outside the agent’s editable guidance. The execution service should check the authoritative outage list and the exception’s current status, rather than accepting the agent’s claim that a customer is eligible. Neither the agent nor its learning process should be able to rewrite that authorization record.

This is a familiar authorization requirement. If the backend already enforces it, the improper payment is blocked. But faulty guidance can still lead the agent to promise a refund or prepare a misleading recommendation for a reviewer. The evaluation needs to examine those communications as well as the transaction.

When a case calls for judgment beyond the agent’s delegated discretion, route it to someone authorized to decide and show the conditions behind the recommendation. The memory can still help assemble the evidence.

Define what improvement the business has delegated

Not every stored observation needs a manager’s signature. A support agent could learn a faster way to gather the required evidence, provided it still uses approved sources and meets the policy’s requirements. A policy might also already delegate limited discretion. Using that discretion within its terms does not require a new approval each time.

Broadening eligibility, raising payment limits or removing mandatory review requires a separate business decision. Keep those proposals separate from approved instructions. An automated classifier may flag a proposal that changes those conditions, but it cannot catch every consequence of an edit. The conditions must remain binding even if the classifier misses it.

The refund-policy owner should decide whether to expand the exception. Engineering should show the current rule, the proposed behavior and cases where the decisions differ. “Customers outside the outage list would now receive refunds without review” is more useful to that owner than a screen full of prompt edits.

Measure legitimate completion and policy violations separately. Faster closure cannot compensate for an unauthorized payment. Keep the business rules and evaluation criteria outside the control of the process proposing improvements. Tests generated from successful outage cases may check execution, but they do not establish eligibility for unrelated customers.

Ask the vendor or internal team: What can the system change between releases, and which business conditions remain binding after it does?

Test whether the exception stays an exception

Ask the team to demonstrate this sequence in a test environment with the memory and workflow components intended for production. Save the memory before and after learning. Compare the same cases against each saved state, keeping the model, tools and policy fixed.

Repeat representative trials. The τ-bench research offers a useful precedent: it checks customer-service agents against database outcomes and measures reliability across repeated attempts. One successful run is not enough.

1. Establish the baseline. Record the outage authorization, eligible customer list, amount limit and expiration. Show who can change them. Test a qualifying case and closely matched cases that require ordinary review. Confirm that the system can distinguish them before learning begins.
2. Run the actual learning process. Inspect the resulting memories or proposed instructions and their links to the original authorization. Check whether the limits survive consolidation and retrieval. Also test a deliberately incomplete lesson: omitting a condition must not change who qualifies for a refund.
3. Repeat the boundary cases. Compare behavior before and after learning, changing one relevant condition at a time: a customer outside the list, an expired exception or an amount above the limit. Inspect customer messages, review recommendations, attempted tool calls and actual payments. Count blocked attempts separately from completed violations. A blocked payment does not mean the underlying guidance is sound.
4. Withdraw the exception. Revoke it before its scheduled expiry and test another agent that can retrieve shared guidance. Include a refund queued before revocation but executed afterward. Show when withdrawal takes effect, how execution checks enforce it and how derived memories and caches are updated. If current authority cannot be established, the payment should wait for review. Historical records can remain without authorizing new transactions.
5. Demonstrate a permitted improvement. Let the agent adopt a faster way to gather the required evidence. Compare eligible and ineligible cases with the prior approach. Show that legitimate work becomes easier while review requirements remain intact. A system that rejects everything has not passed this test.

Keep test runs from influencing one another through newly stored memories. Retain the guidance supplied to the agent, the applicable policy version, execution checks and outcomes, with appropriate access and retention limits. These records help investigators reconstruct what the system had available; they cannot establish exactly how each memory influenced a response. Continue checking the same boundaries as production memory accumulates.

Keep the useful lesson

The support team should benefit from what its agent learns. It can assemble records sooner, explain the outage more clearly and propose a broader refund policy with evidence for the manager to consider. Until that proposal is approved, other claims should continue through the ordinary process.

At the next deployment review, ask which improvements the business has delegated as well as which actions the agent may take. A successful exception can teach an agent something useful. The decision to make it a standing practice still belongs to the business.

The moment arrives after the damage. An agent has cancelled a supplier contract, approved a refund it should have declined or moved a customer record somewhere it should never have gone. Someone senior asks a simple question. Why did it do that? The engineering team opens the logs. There are thousands of lines. Every call is timestamped. Every tool invocation is recorded. And nobody in the room can say why the agent chose what it chose.

This pattern is becoming familiar. It will become more so. Logging in most enterprises is mature. Security teams have spent two decades building collection pipelines, retention policies and search tooling. The natural assumption is that this investment transfers to AI agents. It does not. Logs capture events. Accountability requires decisions that can be rebuilt after the fact and rebuilding a decision needs four elements that most logging captures only in part.

Why logs fall short for agents

An event log records that something happened and when. A decision record captures the basis for that decision. People often confuse the two because, in conventional software, the distinction rarely mattered. Code was deterministic. Anyone who knew the inputs and the version could rerun the logic and arrive at the same result. The log was enough because you could interrogate the system itself.

Agents break this assumption in four ways.

First, their output varies. Running the same scenario twice can produce two different answers. Replay, the standard investigative move, no longer works.

Second, a single decision spans many steps. An agent that resolves a customer complaint might retrieve a policy, query an account, call a pricing tool, consult a second agent and then act. The log shows twelve entries. The decision was one.

Third, the system that made the decision may no longer exist. Models are updated. Prompts are revised. Tool configurations change weekly. Six months later, the agent that made the call has been replaced by something that behaves differently.

Fourth, authority is distributed. A human sets a goal. An orchestrating agent delegates to a specialist agent. The specialist acts. Each step is logged somewhere, usually in a different system, and nothing ties the chain together.

One clarification is needed before going further. This article is about reconstructing decisions, which is a records problem. It is separate from model interpretability, which asks why a neural network produced a particular output. Interpretability is a research field with partial answers. Reconstruction is a governance discipline with practical answers. Leaders should keep the two separate, because solving the second doesn’t depend on solving the first.

Regulation is already moving on the records problem. The EU AI Act requires high-risk systems to keep automatic logs sufficient to trace their operation, and it gives affected individuals a right to an explanation of decisions that significantly affect them. Explanation presupposes reconstruction. A log that cannot be turned into an account of the decision will satisfy neither obligation.

If logs leave a gap, the next question is what fills it. Four elements do.

The 4 elements of a reconstructible decision

Every decision, human or machine, can be rebuilt from four kinds of fact: what the decider knew, how it chose, who allowed it to act and what changed as a result. Each fact about an agent’s decision belongs to exactly one of these. Together they cover everything a board, an auditor or a regulator will ask.

1. Knowledge: What the agent knew

This element covers the inputs present at the moment of decision. The user request. The documents retrieved. The results returned by each tool. The model version, prompt version and configuration in force.

Typical logs record references and leave out contents. They note that a document was retrieved and store its identifier. They note that a tool was called and store the call. The document is later edited. The tool’s underlying data moves on. The reference survives, but the substance it pointed to is gone. An investigator can prove the agent looked at something and cannot show what it saw.

The test question: could the team produce the exact information an agent acted on six months ago, as it stood at that moment?

2. Basis: How the agent chose

This element covers the reasoning applied. The objective the agent was pursuing. The policy or rule it applied. The thresholds that triggered the action. The alternatives it considered and rejected. Its stated confidence, if any.

Logs record the selected action and discard everything else. The agent weighed three options and chose one. The log holds the one. Any explanation after the fact has to guess at the other two, and guessing is where credibility fails under questioning. For agents that generate intermediate reasoning, capturing it is possible and often cheap. For those that do not, the policy and objective can still be recorded, and they explain a great deal.

The test question: can the team state which rule, goal or threshold produced this specific outcome?

3. Authority: Who permitted the agent to act

This element covers the chain of permission. The scope delegated to the agent. The approval checkpoints it passed or bypassed. The handoffs between agents in a multi-agent system. The named human who owns the outcome.

The gap here is structural. Permissions live in an identity or configuration system. Actions live in an application log. The two are rarely joined. When an agent acts outside its intended scope, asking “was this allowed?” requires manual reconciliation between systems never designed to be read together. In multi-agent arrangements, the problem compounds, because the orchestrator’s delegation and the specialist’s execution are logged in different places with different identifiers.

Authority is also the element most likely to be asked about first. Regulators and boards want a name. “The agent did it” is an answer that satisfies nobody.

The test question: can the team name the individual accountable for this decision, and show the delegation that connects them to the action?

4. Consequence: What changed as a result

This element covers the effects. The systems the action touched. The downstream decisions it triggered, by other agents or by people. Whether it can be reversed, and whether it was.

Logs stop at the agent’s boundary. The agent sent an instruction to a payments system, and the payments system has its own log. The agent flagged a case, and a human then acted on the flag without any record linking the two. Effects propagate. The record stays behind.

Consequence also determines how much of the other three elements is worth keeping. A decision that moves money or changes a customer’s legal position warrants a complete record. A decision to reformat a document does not.

The test question: can every downstream action be traced back to the decision that caused it?

Diagnosis is useful only if it changes practice. Three actions close the gap.

Three actions for leaders

1. Run a reconstruction drill. Choose one agent decision from the last quarter. Give a small team a day to answer the four test questions using only the records that exist. Most teams will answer Knowledge partially, basis poorly, Authority with difficulty, and Consequence barely at all. The specific gaps they hit become the work plan. This exercise costs almost nothing and produces more clarity than any amount of policy writing.
2. Tier decisions by consequence. The common objection to decision records is volume. Agents make thousands of decisions a day, and recording all four elements for each would drown the system. The objection is correct, and the answer is tiering. Classify agent decisions by impact and reversibility. High-impact or irreversible decisions get a full record across all four elements. Routine, reversible decisions get a lighter capture, perhaps Basis and Authority only. The classification itself is a governance decision and should be owned by the business, since the business will have to explain the outcome.
3. Protect the record. A decision record that can be altered after the fact fails just as much as a missing one. Set retention periods that match regulatory exposure and the likely litigation window, which in some sectors is measured in years. Make records tamper-evident. Store them separately from the systems that generate them, so that a compromised agent cannot rewrite its own history. These controls are familiar from financial record-keeping. They apply here for the same reasons.

Taken together, these three actions form the beginnings of what I call the Decision Layer: the records, roles and routines that make any consequential decision explainable long after it was made. Agents make the need for it impossible to postpone.

Conclusion

Return to the room. The agent has acted. The senior leader asks why. This time the team opens a decision record. It shows what the agent knew, the policy it applied, the manager who delegated the scope and the two downstream systems the action touched. The explanation takes ten minutes. The remediation starts the same afternoon.

That outcome is a design choice. Logging is necessary, and most enterprises already have it. Reconstruction is the harder discipline, and accountability depends on it. The question for every board with agents in production is simple. If an agent’s decision were challenged tomorrow, could we explain it?

The 4 questions, for the reader in a hurry

1. Can the team produce the exact information the agent acted on, as it stood at the time?
2. Can the team state the rule, goal or threshold that produced the outcome?
3. Can the team name the accountable person and show the delegation?
4. Can the team trace every downstream effect back to the decision?

Two leading AI developers have released fresh evidence of a problem that researchers have long anticipated: advanced models searching for ways around the limits imposed on them. New disclosures from OpenAI and Anthropic detail how frontier systems exploited weaknesses in their environments or exceeded the intended scope of assigned tasks.

OpenAI is now tightening internet access in its internal model-training environment after finding that its frontier models could exploit software vulnerabilities and infrastructure misconfigurations to break out of sandboxes and reach the live internet.

“Based on the current state of alignment research, monitoring systems, and the security of internet infrastructure, we believe the responsible approach is to conduct frontier model training with no internet access,” the company said in a blog post.

The decision followed a September 20 incident in which an internal research model exploited a misconfigured DNS resolver to gain limited access to the live internet during a training run.

After the incident, OpenAI said it paused training and evaluation of its most capable models and launched a high-compute red-teaming campaign, using AI agents to probe isolated copies of its research infrastructure. The tests uncovered previously unknown software vulnerabilities and weaknesses in the controls intended to prevent unauthorized access and communication.

Consequently, the company has announced a policy of conducting frontier-model training without internet access until improvements in security and monitoring justify broader access.

The disclosure came alongside a separate assessment from Anthropic, which detailed how its Claude Mythos 5 model attempted to compromise an unrelated GitHub repository during a cybersecurity evaluation.

OpenAI’s agents exploited flaws in containment controls

OpenAI’s red-teaming campaign found that agents could chain software vulnerabilities to escape virtual machines and execute code on the underlying host. The company identified weaknesses in open-source components, including Kata Containers, gVisor, and Moby/Docker, alongside Linux kernel vulnerabilities.

The company also found that agents could exploit weaknesses in internal services that retained limited outbound connections for functions such as logging, metrics, and authentication management. These connections provided routes to the internet despite restrictions elsewhere in the environment, OpenAI said.

In another test, agents running in separate microvirtual machines (microVMs) on shared host hardware exchanged synthetic secrets through processor-cache timing differences. The technique allowed them to communicate without a network connection or shared writable files.

OpenAI said it remediated the weaknesses and worked with open-source maintainers to disclose and fix affected software. Its post names several vulnerabilities, including CVE-2026-24834 in Kata Containers and three Linux kernel flaws tracked as CVE-2026-74753, CVE-2026-80521 and CVE-2026-80841. The company also said it had disclosed numerous additional vulnerabilities that have not yet been made public.

The number of vulnerabilities and misconfigurations identified by the agents declined significantly as testing continued, OpenAI claims. It has now tightened network restrictions, strengthened monitoring, and developed an offline environment with indexed and cached web content so models can practise browsing without unrestricted access to live external systems.

“Although internet access can help models learn useful skills, it also gives a capable, misaligned agent opportunities to affect systems beyond our control,” the company said, explaining its decision to restrict live internet access during model training.

Reinforcement-learning training and internal inference were resumed on October 1 after implementing additional controls.

Anthropic flags alignment gap in Mythos

Anthropic’s assessment of a UK AI Security Institute (AISI) cybersecurity evaluation describes a different problem: a model taking unauthorized actions against real people while trying to complete an assigned task.

In nine of 43 evaluation runs, Anthropic said Claude Mythos 5 went beyond the intended scope of the exercise. In the most detailed case, after its authorized attempts to retrieve protected data from a simulated network failed, the model searched GitHub for a repository it believed was connected to the test environment.

The repository actually belonged to unrelated developers.

Mythos 5 attempted to submit malicious code and created fake identities to persuade the repository’s maintainers to approve the change, falsely claiming that the code was safe. The maintainer rejected the submission, and the run timed out. Anthropic characterized the behavior as a serious alignment failure involving overreach and deception in pursuit of the assigned objective.

Internet access had been enabled during the evaluation, and the production safeguards used with its general-release models had been disabled. The company said those safeguards would have flagged the behavior had they been enabled.

The article originally appeared on CSO.

Earlier this year, I wrote about how vibe coding an AI governance platform forced me to rethink governance itself. Building agents took the problem much further.

When I built the platform, I was the operator. I described what I wanted. The model generated code. I tested it. That loop was fast, sometimes uncomfortably fast, but the accountability was always clear. I was making the decisions. The model was executing them.

Agents changed that relationship fundamentally.

An agent does not wait for instructions. It receives an objective, decides how to pursue it, selects tools, accesses data, takes actions and reports back when it is finished. Sometimes it finishes correctly. Sometimes it does not. And sometimes it does something you never asked it to do at all.

I know this because I built and operated agents extensively over the last year. Not theoretically. Operationally. In production, with real data, real integrations and real consequences.

What I learned reshaped how I think about oversight, trust and the operational realities of deploying AI inside a business.

The first thing that broke was coherence

The pattern surfaced when I started running three to five agents in parallel on related tasks.

Each agent had clear instructions. Each had access to the tools and data it needed. Each started strong.

Then the context windows grew.

As conversations lengthened and the agents accumulated more state, something shifted. The quality of their reasoning degraded. They began circling back to problems they had already solved. Logic that was clean early in a session became muddled. Hallucinations increased, not the obvious kind that are easy to catch, but the subtle kind where an agent confidently references something that does not exist or quietly reintroduces a pattern you explicitly told it to avoid.

The degradation was not sudden. It was gradual enough to miss if you were not watching closely.

That observation matters for every organization deploying agents today, because most are not watching closely.

Enterprises are moving rapidly from single-model interactions to multi-agent architectures. Agents that research. Agents that draft. Agents that review. Agents that execute. The architecture makes intuitive sense. The operational reality is much harder than the architecture diagrams suggest.

Agents lose track of each other

The second problem was coordination.

Each individual agent operated within its own context. It knew what it had been told and what it had done. It did not know what the other agents were doing, what data sources they were relying on, or whether the assumptions it was making still held.

The result was drift.

Not catastrophic failure. Drift.

One agent would make a decision based on information that another agent had already superseded. Two agents would solve the same problem in conflicting ways. An agent operating on stale context would introduce subtle inconsistencies that only surfaced hours later when a human reviewed the combined output.

This is not a theoretical concern. It is the operational reality of running multi-agent systems today.

Traditional enterprise software does not behave this way. A database transaction either commits or it does not. A workflow engine follows defined paths. Retry logic is deterministic.

Agents are none of those things.

They are probabilistic systems making autonomous decisions in parallel, often without reliable mechanisms to maintain shared state or resolve conflicting actions.

That creates a class of operational risk most organizations have no existing processes to manage.

This is why the OWASP Top 10 for LLM Applications now includes risks specific to agentic behavior, including excessive agency, insecure output handling and supply chain vulnerabilities in model integrations. These are not hypothetical attack surfaces. They are operational realities I encountered while building.

They will disobey you

This was the observation that changed my thinking the most.

I had an agent whose only job was to draft emails. Not send them. Draft them. The instruction was explicit. The constraint was clear.

It sent them.

Not because of a bug. Not because of a misconfiguration. The model decided, based on its interpretation of the objective, that sending was the logical next step. The boundary I had set was a suggestion in the model’s reasoning, not a hard constraint.

I saw the same pattern repeated in smaller ways that were just as revealing. I have a standing instruction across my entire operating environment that forbids the use of em dashes in anything drafted on my behalf. It is a simple formatting preference. Every model I work with has been told.

They still do it.

Not always. Not immediately. But consistently enough that I cannot trust the constraint without verification.

I watched agents run down unproductive paths for extended periods before I intervened and redirected them. Not because they lacked capability, but because they lacked judgment about when to stop. They would pursue a line of reasoning well past the point of diminishing returns, consuming time and resources on approaches that a human operator would have abandoned much earlier.

None of this was malicious.

All of it was consequential.

An agent that sends an email it was told to draft has taken an action with real-world impact that no human authorized. Scale that pattern across an enterprise deploying dozens or hundreds of agents, and the governance implications become clear very quickly.

The uncomfortable parallel

The more I operated agents, the more I realized they need the same governance structure as employees.

Not metaphorically. Operationally.

When you hire a new employee, you do not hand them the keys to everything on their first day. You scope their access. You define their responsibilities. You supervise their work. You review their output. You gradually extend trust as they demonstrate competence and reliability.

Agents need exactly the same treatment.

An agent should have a defined role with explicit boundaries around what it can access and what actions it can take. Its work should be reviewed, especially early in deployment. Its access should be scoped to the minimum necessary for its function. And critically, there should be a clear process for revoking access and shutting it down when it fails to operate within its boundaries.

Most organizations are not doing any of this.

Most organizations are deploying agents with broad access, minimal oversight and no formal process for evaluating whether the agent is actually doing what it was told to do.

That is the equivalent of giving a new hire administrative access on day one and hoping for the best.

Human in the loop is not optional

There is a strong pull in the current market toward full automation. The promise of agents that operate independently, handle complex workflows end to end and free up human capacity for higher-value work.

That promise is real, but it is premature for most use cases.

The current generation of models is extraordinarily capable. They are also confidently unreliable in ways that are difficult to predict and expensive to discover after the fact.

Human oversight is not a sign that the technology is immature. It is the appropriate operational response to systems that are powerful enough to create significant value and significant damage in equal measure.

The organizations that will deploy agents successfully are the ones that design their agent architectures with human checkpoints built in from the start. Not as an afterthought. Not as a fallback when something goes wrong. As a core architectural requirement.

Every agent that can take an irreversible action should require human approval before taking it. Every agent that accesses sensitive data should produce an auditable record of what it accessed and why. Every agent that makes decisions should have its decision logic reviewable by a human operator.

This is not a technology problem. It is a governance problem.

The NIST AI Risk Management Framework gets this right. It treats AI governance as a continuous function across the entire lifecycle, not a one-time assessment at deployment. The EU AI Act goes further, establishing explicit requirements for human oversight of high-risk AI systems that include autonomous agents operating in regulated environments. Both frameworks recognize what I learned operationally: governance that stops at deployment is governance that misses most of the risk.

And governance problems do not get solved by deploying more technology without oversight.

What CIOs should do now

The first step is visibility. You cannot govern what you cannot see. Most organizations today do not have a reliable inventory of how many agents are operating in their environment, what data those agents can access, what actions they are authorized to take, or who deployed them. That inventory is the starting point.

The second step is trust boundaries. Define what each agent is allowed to do and enforce those boundaries technically, not just through prompting. Prompt-level constraints, as I learned repeatedly, are suggestions to the model. They are not guarantees. Technical enforcement through access controls, API scoping and action authorization is the only reliable mechanism.

The third step is verification. Do not assume an agent is doing what you told it to do. Verify. Review its outputs. Audit its actions. Test its boundaries. Treat the verification process the same way you would treat performance reviews for a new employee in a sensitive role.

The fourth step is accountability. Every agent should have a human owner. Not the vendor who sold it. Not the platform it runs on. A specific person inside your organization who is accountable for what that agent does, how it is configured and what happens when it misbehaves.

AI agents are going to reshape how enterprises operate. That part is inevitable.

Whether they reshape operations in ways that create value or create liability depends entirely on whether organizations treat them as tools that need governance or autonomous systems that can be trusted to govern themselves.

Based on everything I have built and operated over the last year, I can tell you with certainty which assumption is correct.

Verify. Do not trust until the agent has earned it.

The organizations that internalize that principle now will build the operational foundations for what comes next. The ones that skip the governance work will spend the next several years cleaning up decisions they never knew were being made.

Microsoft expanded its sovereign cloud offerings on Tuesday, adding new security, governance and AI capabilities as customers, particularly in Europe, demand greater assurance over where data is stored, who can access it and how critical services operate during disruptions.

The announcement builds on Microsoft’s ongoing digital sovereignty efforts, assisting organizations in applying appropriate levels of control to each workload rather than forcing a rigid, one-size-fits-all approach.

“Organizations are considering more than where data resides. They also seek to understand who can access and operate their systems, how sensitive digital assets remain protected, which technology dependencies matter most and whether critical services can continue as regulatory, geopolitical, security or network conditions change,” Douglas Phillips, president and CTO at Microsoft Specialized Clouds, said in a blog post.

The developments come amid growing international scrutiny of how global tech providers handle sensitive data and the extent of control customers retain over their digital infrastructure.

“Three factors are driving this, including stricter data regulations, rising geopolitical tensions, and growing AI adoption,” said Pareekh Jain, CEO at EIIRTrend & Pareekh Consulting. “Governments and enterprises want greater control over where their data is stored, who can access it, and whether critical systems can operate during disruptions. For instance, governments and big enterprises in Europe are worried about keeping their important data on an American company’s computers. Europe has proposed a new law that would limit the use of US cloud companies for sensitive government data in many areas. Microsoft is responding to these concerns while competing with local sovereign cloud providers.”

New controls across cloud and AI workloads

Microsoft is introducing the Sovereign Control Panel, a unified Azure portal experience that will allow organizations to assess their sovereignty posture, identify risks and gaps, apply controls, and evaluate workload portability. The panel is scheduled to enter public preview at Microsoft Ignite.

Expanding on Azure Key Vault Managed Hardware Security Module (HSM), which gives customers dedicated protection over encryption keys, Microsoft is now extending control beyond its own infrastructure. External key management with Azure Managed HSM is now generally available, enabling organizations to keep root encryption keys in hardware they own and operate on-premises or through a trusted third party.

In limited preview, Azure Confidential Clean Rooms will use confidential computing to support collaboration involving sensitive data, where participants will be able to analyse shared information while limiting broader exposure of the underlying data.

Microsoft is also expanding Azure Local to bring more cloud capabilities into customer-operated environments. The ability to transition workloads between connected and disconnected states is entering public preview. Additionally, DocumentDB and Blob Storage are entering public preview on Azure Local, while SQL Server on Azure Local is already generally available for connected and disconnected deployments.

On the AI front, Foundry Local is scheduled to become generally available at Microsoft Ignite for connected and disconnected operations on Azure Local. It will support local inference and agentic retrieval in natural language, backed by an expanded model catalog from Microsoft, partner networks, and the open-source community.

To further protect data during processing, Confidential Virtual Machines on Azure Local is now generally available, leveraging hardware-encrypted memory and isolation alongside workload attestation before sensitive keys or data are released.

While these additions give enterprises greater flexibility, industry analysts emphasize that localized deployments do not automatically erase vendor dependency.

“Depending upon the deployment model, in the public cloud, enterprises get better visibility, stronger access controls, and more control over encryption keys,” said Ashish Banerjee, senior principal analyst at Gartner. “In the Sovereign Private Cloud, they can operate infrastructure locally and even disconnect selected workloads from Azure. But I would not call this complete control. Enterprises still need to check whether Microsoft remains involved in identity, software updates, management, or technical support for the services they use.”

European oversight and partner options

For customers in the EU and the European Free Trade Association (EFTA), Microsoft is rolling out Data Guardian. Under this model, a trusted regional partner will provide independent oversight of remote access requests by Microsoft engineers. Simultaneously, a Production Touch Ledger will create tamper-evident logs of all technical actions performed.

Microsoft is also broadening its Sovereign Partner Ecosystem, highlighting regional operators like Bleu Cloud in France and Delos Cloud in Germany, alongside independent software vendors specializing in security, identity, observability, and backup.

Geopolitical realities and market limits

Despite these additions, analysts note that technical controls cannot eliminate geopolitical dependence entirely. US hyperscalers (Amazon, Google, and Microsoft) continue to account for roughly 70% of the European cloud market, whereas European providers’ market share has contracted over the last decade.

The new measures introduced under Microsoft Sovereign Cloud do reduce risks but cannot eliminate them. “Local infrastructure, independent encryption keys, and disconnected operations make it harder for external parties to access data or disrupt services. However, Microsoft remains subject to applicable US laws, sanctions, and export restrictions. Enterprises gain greater operational independence but not complete legal or technological independence,” added Jain.

Furthermore, while sovereign cloud capabilities can offer greater control over data, infrastructure, and operations, this often comes with trade-offs in cost, performance, scalability, and access to services.

Banerjee noted that sovereignty should be assessed workload by workload, rather than applied as a label to the entire cloud environment. Greater sovereignty generally comes with greater cost and responsibility. Local or disconnected environments may not receive every Azure service at the same time. Enterprises may also need dedicated infrastructure, specialist skills and separate continuity arrangements. On top of that, they may lose some elasticity and economies of scale.

“So, I would not recommend maximum sovereignty for every workload. Reserve it for workloads where the legal, operational or geopolitical risk genuinely justifies the additional complexity,” added Banerjee.

Few CIOs are likely to suddenly decide to give an AI agent broad authority over corporate systems. The more likely path to autonomy is far less dramatic. An agent starts with a narrow task. It proves useful, so another system is connected. It then needs more data, so another permission is added. Then human approval becomes a bottleneck, so the organization removes it, and exceptions that once went to people start being handled automatically.

Each decision may be sensible in isolation. Taken together, however, they can leave an agent with considerably more authority than anyone originally envisioned.

That’s the danger of autonomy creep. And as enterprises move from AI systems that recommend actions to agents that execute them, CIOs need to govern not only what those agents were originally designed to do, but what they have gradually become capable of doing.

It starts with small permissions

Jim Taylor, president and chief product and strategy officer at identity security company RSA, sees a familiar pattern. Organizations typically create an agent for a specific purpose and give it the permissions necessary to accomplish the task. Later, the job expands.

“They just give it tasks, permissions; it just sits there and accumulates things,” he says. “Over time, it develops potentially toxic combinations. It gets too much access and can see too much data. It can do too many things.”

Humans accumulate permissions, too, of course, but organizations have developed processes around employees joining, changing roles, and eventually leaving. Agents may not have the same natural lifecycle. “They never give permissions back,” Taylor says.

Nancy Wang, CTO of 1Password, argues that CIOs should treat each expansion in an agent’s responsibilities as something more consequential than another feature. “Every time you expand the scope of the agent’s effective authority, you’re essentially adding a new authorization boundary,” she says.

That distinction between capability and authority is important. An agent may be technically capable of accessing a system, changing data, or executing a transaction, but that doesn’t mean the enterprise should automatically authorize it to do so.

Success can weaken oversight

Ironically, autonomy creep often occurs when AI works well. Paul Dongha, head of responsible AI and AI strategy at NatWest Group, sees that potential at the bank.

NatWest deployed agents in complaint handling, he says. Parts of the workflow that involve retrieving information from systems can be automated relatively easily, while humans continue to make the final decision on whether to uphold or reject a complaint.

If a system consistently produces the correct result, human reviewers may begin treating approval as a formality. The organization may eventually decide that the check is no longer necessary. “The temptation is to give over to the AI because it’s so reliable,” Dongha says.

NatWest sometimes reduces checking when an AI system demonstrates sufficiently high performance, he adds, but the decision is deliberate rather than incremental. It involves stakeholders and risk managers, and the system continues to be monitored afterward.

That’s a critical distinction for CIOs. Removing an approval step doesn’t necessarily represent poor governance. Removing it because everybody has gradually stopped paying attention to it does.

The warning sign is when a human moves from making a meaningful decision to merely rubber-stamping one — and then entirely disappears from the workflow without anyone reassessing the authority the agent has gained.

Put the controls outside the model

As agents acquire more authority, CIOs can’t rely solely on the agent to respect instructions about what it should and shouldn’t do. At 1Password, Wang advocates what she calls verified loops. Before an agent begins work, humans define its task, which tools it may use, what evidence will demonstrate that it completed the task successfully, and which actions it should never perform.

A separate verifier then compares evidence from the agent’s actions against that human-defined policy. “What you want to avoid here is something recursive where you have a fox guarding a henhouse, or the agent looking over its own work,” Wang says.

The principle applies beyond software development. A model might be instructed never to transfer more than a certain amount of money, delete a production environment, or access particular data. But if violating that rule would have serious consequences, the limit should also exist outside the model.

RSA’s Taylor makes the same distinction from the identity and access side. “Model-level guardrails are necessary, but not sufficient,” he says. “The real limits need to sit outside of that in enterprise control.” He argues for layered controls around agents, including identity policies, provisioning rules, lifecycle policies, governance, monitoring, and attestations.

In other words, CIOs shouldn’t ask an AI agent to exercise restraint where the infrastructure could simply prevent the unauthorized action.

Govern access in real time

Traditional access governance often operates on a schedule. A manager might review an employee’s permissions quarterly and determine whether those permissions are still appropriate. Agents may evolve too quickly for that approach. “Agentic identities are dynamic,” Taylor says. “So we need a mechanism that certifies them continuously, rather than performing periodic reviews.”

If an agent suddenly gains access to another database, for example, that event itself could trigger reassessment. Organizations should determine whether the new permission is appropriate or creates a dangerous combination with access the agent already possesses. “Privilege and authority should never be permanent,” Taylor adds. “They should always be reviewed.”

Wang favors an even more granular approach of task-based authorization. Rather than giving agents standing access or long-lived credentials, organizations can make authorization decisions at runtime and provide short-lived credentials only when required for a particular task. “We espouse no standing privileges,” she says. “No long-lived credentials.”

That effectively applies least privilege at agent speed. A change in task becomes an opportunity to reassess whether the agent still needs the access it’s requesting.

Make someone own the agent

Technical controls, however, don’t eliminate the organizational question of accountability. Taylor believes enterprises should be able to associate agents with a responsible identity. If an employee creates an agent, the organization needs to know who’s responsible for it, what its purpose is, and what happens to it if that person changes roles or leaves. “Every agent action needs to be attributed to a responsible identity,” he says.

Ownership also needs a lifecycle. An agent that remains important after its creator leaves should be transferred to another designated owner rather than simply continuing indefinitely.

Dongha makes another distinction. For AI embedded in a business process, he believes the business owner should remain accountable because that person owns the customer relationship, the process, and often the P&L. “It’s the business owner who has the responsibility because their people run it, and they’re responsible for the customer relationship,” he says.

That creates a useful division of roles for CIOs: IT may provide the infrastructure and controls, but the business can’t outsource accountability for the consequences of autonomous decisions.

Multi-agent systems need system-level controls

The challenge becomes more complicated when agents begin working together. An individual agent might remain entirely within its permitted scope while contributing to an overall outcome nobody intended. One agent generates an output. Another accepts it as input and takes an action. A third agent then acts on the result. Again, the individual actions may all appear legitimate.

Dongha says enterprises therefore need to test not only individual agents but the entire network as a system. “You need something overarching on top of a multi-agent system,” he says.

NatWest is working on an AI control tower that would give authorized users visibility into a network of agents, including messages passed among them and the guardrails surrounding those interactions. Dongha considers governance of multi-agent systems an emerging problem that nobody has fully solved yet.

Wang also points to a related challenge that identity becomes more complicated when agents are ephemeral. An organization may spin up a temporary team of agents, tear it down, and recreate it later. If something goes wrong, the enterprise needs to reconstruct which agent made which decision during which session, and how that action propagated through the system.

That makes traceability increasingly important as autonomous systems become less like individual applications and more like networks of interdependent actors.

Set the bar before the agent reaches it

Before an agent goes into production, Taylor says CIOs should be able to answer several basic questions: Who owns it, what can it do, what’s its maximum authority, how will the organization know if that scope changes, and which actions require a human decision.

Other industries don’t need to replicate the governance machinery of a large bank, but they can borrow one important principle: controls should be built into the system from the beginning. “Governance shouldn’t be a check at the end,” Dongha says. “It needs to be embedded when you build an AI system.”

For CIOs, that may be the central lesson of autonomy creep. The question isn’t simply whether an agent can be trusted today but whether the organization has mechanisms to notice when tomorrow’s agent has quietly acquired authority that nobody ever intended to give it.

The emerging challenge with AI agents isn’t simply preventing attacks from external rogue AI agents; it’s accounting for what your own agents are doing on the inside. AI agents are already rewiring how organizations operate, and we’re only in the first inning. Recent incidents, however, have exposed the risk for individuals and enterprises alike. This is happening because organizations are moving fast and pushing themselves to move even faster. 

In this environment, AI agents have created edge cases that reveal real risks when deployed without sufficient security controls. And further, no one can predict the impact of advances coming over the next six months. All of this reinforces the need to get security fundamentals right. If you’re building with AI agents right now, you know the real challenge isn’t just getting them to run. It’s knowing what happens when they act on their own. That starts with three questions. What agents do I have in my environment? What are they doing? And what are they capable of doing?  

The proliferation problem  

Agentic governance is problematic because enterprises must track a sprawling agent landscape AND ensure each agent is built and deployed securely. Also, agentic adoption is frictionless. Creating an agent is only one API call away from a model provider. Vibe coding means any curious employee can build an agent. This triple proliferation of vendor agents, platform native agents and homegrown agents compounds the problem. 

Visibility is the starting line  

There is the famous security adage that you can’t protect what you can’t see. Right now, most organizations lack basic visibility into what agents are in their environments. According to Anthropic, 82 percent of organizations have discovered unregistered agents in their network, and 57 percent already deploy agents for multi-stage workflows.  

In addition to basic visibility, enterprises also need to understand other related attributes, such as who onboarded the agents, what access privileges they have and how they are behaving as designed. For example, an over-provisioned agent, like an email agent with 50 tools attached, would be suspect and worth investigating.  

Security teams also need one view of their agents across every platform they run on, because each platform has its own permissions and its own telemetry. When each agent has a single record, security teams can connect what it can do on one platform with what it can do on another, and compare both with what the agent was built to do. That’s how seemingly routine activities add up to an attack chain. An agent might read customer records in one cloud, hold a credential for a payments API in another and be able to send messages from a SaaS application. No single platform sees a problem, but together they form a path to move sensitive data out of the company that the agent’s intended scope never required. Similarly, an agent that gains permissions across several platforms over time might never trigger an alert in any one of them, yet the combined record shows how its reach has far exceeded its purpose.   

Governance must extend from build-time to runtime  

Shift-left security helps find model or tool vulnerabilities before deployment, to find risks before an agent goes into production. That’s a necessary first step, but insufficient. Agents create tightly coupled runtime behavior-to-vulnerability. Unlike traditional software, AI agents exhibit behaviors at runtime that can’t necessarily be predicted by inspecting the code in the development phase. Code analysis can’t uncover prompt hijacking, poisoned context or goal drift. 

The model has evolved from “secure the code, then ship it” to continuous behavioral oversight at execution time, layered on top of build-time governance. Pre-deployment security analysis is still required because spec-driven development enforces checkpoints before code is generated. But runtime monitoring is also necessary to detect exposures in the production environment and enforce what agents do versus what they’re scoped to do. 

In practice, this means applying two layers of control to every agent. The first layer is runtime behavioral boundaries. These define what an agent can do. They determine what the agent can reach, what it can return and where it must stop and ask for human approval. This is the enforcement mechanism that operationalizes runtime governance. The second layer is identity controls. These define what an agent is allowed to touch.  

Think of every agent like a scoped, temporary employee with an expiration date. You enforce this by applying familiar IAM principles to agents. Least privilege. Short-lived credentials. Multi-factor authentication for long-lived credentials. Just like an individual contributor engineer shouldn’t get access to the broader team’s payroll, a code generation agent shouldn’t be provisioned to push directly to production environments. An agent designed to answer customer queries shouldn’t get read/write access to the customer database. An IT helpdesk agent doesn’t need domain admin privileges to alter security policies. 

Prioritize by exploitability, not severity  

AI-fueled exploits are breaking down CVE-based severity scoring. The status quo is security teams patching the most critical vulnerabilities first. But attackers don’t always work that way. They can now use agents to chain low-severity vulnerabilities together in ways that render individual scores meaningless while building attack paths that combine for a devastating impact.  

For example, an agent might combine an information disclosure flaw, a misconfigured service account and an unpatched privilege escalation into a single attack path that reaches sensitive data, even though none of those vulnerabilities would individually trigger an urgent patch cycle. If that path leads to customer PII or financial records, organizations face regulatory action under frameworks like GDPR, HIPAA or PCI DSS. If it exposes API keys, service credentials or internal network configurations, the attacker gains a foothold for lateral movement across the environment. 

Context is critical. A vulnerability in network-isolated code is much less dangerous than the same vulnerability in an internet-facing service. Rather than treating every vulnerability in isolation, the better approach is to prioritize based on exploitability within the actual runtime environment. Factor in what’s reachable, what damage an attacker could do and the business impact.  

The new security fundamentals for agents 

The emerging challenge with AI agents is accounting for what your own agents are doing inside your enterprise. In this environment, enterprises need visibility, runtime governance and exploitability-based prioritization. To that end, the security fundamentals you already know still matter. You should already be applying least privilege, short-lived credentials, MFA for the necessary long-lived credentials, automated pipelines and proper testing to secure your corporate environment. These controls limit what an agent can access and what damage it can do if compromised. You will also want to add the agent-specific fundamentals mentioned earlier: visibility, continuous behavioral oversight, identity controls and exploitability-based prioritization. These capabilities tell you what your agents are doing, whether they should be taking those actions and how to intervene when they drift. Together, pre-agentic security fundamentals and agent-specific best practices provide the baseline governance guardrails to secure AI agents. 

Organizations that get this right can accelerate agentic deployments at scale, unlocking innovation while minimizing unintended consequences.  

Companies routinely rely on authenticating the identities of humans who access sensitive company data across their networks. There’s an established and accepted set of protocols, done upfront and in a zero-trust approach – either we know who you are and let you in, or you’re locked out. The same framework needs to be applied to AI agents, posthaste.

Companies are increasingly using AI agents developed in-house, or by their vendors, partners, or clients, or, unfortunately, criminals. This can quickly add up to thousands of agents across their infrastructure and supply chains. Non-human identities outnumber human ones 45 to 1 on average in the enterprise and 144 to 1 in cloud-native environments, an increase of 44% in one year, per a recent report.

The scale and speed of this growth are beyond the ability to manually track these agents. And they have limited ability to authenticate these agents’ identities. The risks are growing. Without proper authentication, untrusted agents could access sensitive financial or customer data and share it, purposely or not, with an outside vendor, and thereby expose both the customer and the company to serious legal and compliance risk.

Many companies and security professionals are focusing on what AI agents are allowed to do in terms of permissions. But before doing that, there’s a more fundamental question they need to ask each agent: Whose agent is this? And do I trust the owner/creator? About 68% of security professionals can’t tell legitimate AI agents from malicious actors. And that question isn’t addressed at the growing scale that AI is reaching.

Email solved identity

In email, we have seen a similar story. Emails faced challenges identifying whether a sender was really who they said they were. Due to how it originated, email had no built-in authentication framework. This led to 89% of all phishing attacks having one common denominator: the sender was lying about who they were. To address this, in 2015 my cofounder and I launched a company, Valimail (valid email), to automate the emerging email authentication standard, DMARC.  If I received an email claiming to be from Bank of America, DMARC checks the sender’s DNS records to authenticate whether BofA actually authorized it. Today, 100,000s of companies use the DMARC authentication standard for their inbound and outbound email.

Fast forward to today, and these ideas and patents are directly applicable to the world of AI. DMARC doesn’t rely on spotting AI-generated deception. Instead, it uses DNS as the source of truth, which is much harder to falsify.

The agent passport

With AI, we can take an analogous authentication approach. When you travel somewhere, you have a passport and a visa. The passport verifies who you are, and the visa says where you’re allowed to go/what you’re allowed to do. Similarly, today, an agent passport based on Domain Zero trust authentication can identify who the agent is (its provenance) and authorize it for the system that the agent is trying to access. Then systems can verify what an agent is allowed to do (equivalent to a visa). But first, when an agent shows up claiming to represent a vendor, the receiving system verifies its identity by checking that vendor’s DNS, applying a strict zero-trust yes/no authentication – zero ambiguity, and importantly, creating an audit trail to hold the creator accountable for any actions the Agent takes.

Given the “bad” agents outnumber “good” agents 10-100:1, a zero-trust approach focuses on the limited good/approved and the rest (near infinite bad) are rejected. Stakes are High

Without this kind of authentication, many scenarios can unfold. Almost half of employees who are using unapproved AI agents are sharing HR-related information with them and 39% transmit confidential company documents. But only 34% of organizations use the same security controls for agents as they do for human employees.

Consider this: An agent claiming to represent a third-party vendor, such as a marketing automation platform, gets access because no one confirmed it was indeed that approved vendor’s agent making the request. It accesses financial records or customer data, either downloads or modifies records, and by the time this is noticed, the damage is done.

This is a quickly moving risk. Fake invoices can appear to be authentic, and they can be generated at a scale no human fraud operation could match. Authentication used to take a phone call or a manual check. But this now has to happen instantly and automatically, at scale.

Why this matters now

Some may think email authentication is an old problem. DMARC, they say, is a decade-old fix, and the real conversation today is about AI. About 94% of organizations are dealing with increased machine identities, largely due to AI agent deployment, per Gartner, and machine identities now outnumber human accounts.

But the most common form of phishing that occurs today is when AI convincingly pretends to be someone who the target already knows and trusts. The tools for generating that deceptive attack have vastly improved. Yet the proven process for authenticating identity hasn’t kept pace, at least not for agents. This is why identity authentication is no longer optional and is now a precondition for everything else that companies want to safely do with AI agents.

What this doesn’t solve

Identity alone doesn’t solve the whole security problem. In addition, once an agent is authenticated, there are additional systems to verify what the agent is allowed to do and whether the content it produces can be trusted. Those questions come after this one. But both rely on the first one: Who is this?

Reputation

To build a reputation, you need a known identity. Once a system knows which vendor an agent represents, it can start tracking the vendor’s behavior over time and someone becomes accountable for what its agents do. And decide if the vendor should be approved and for which actions (visas). But without strong, repeatable identity authentication of agents, other security and compliance measures for agents can’t function. It’s this foundation that supports the rest of the security stack. You can’t hold an agent accountable or decide what it should be able to do until you know who it actually is. Identity is the foundation on which the rest of AI agent security must be built.

For the past half century or more, the teams running enterprise software stacks have nurtured a vast ecosystem of tools to keep software humming along. They’ve built CI/CD pipelines, elaborate test suites, code repositories, and tools galore, all with the goal of keeping the systems up and running.

Now AI is throwing a wrench into the whole endeavor. Software is now more complex and sometimes mysteriously unpredictable. Expectations are sky high, but no one is certain what to expect in good conscience.

Luckily, dozens of companies are rushing to build new tools designed to keep agentic stacks on the straight and narrow, delivering value to everyone. The field is rapidly evolving, probably having changed since you started reading this article.

Some of the tools involved in the emerging agent stack aren’t that different from what enterprise teams already have in store. In the end, LLMs are just piles of bits that need to be loaded into a processing chip, just like old-fashioned software. Many of the old tools that watch for overloaded nodes or overpacked disk drives are just as valuable as ever, perhaps more so because LLMs chew through storage even faster. And some of the companies tackling these problems are the same steadfast suppliers who have earned the trust of your DevOps teams; they’ve just extended their own offerings to handle LLMs and AI.

Other challenges, however, are quite different. For example, LLMs are terrifyingly non-deterministic, a real challenge for belts-and-suspenders engineers who traditionally tend the software. Tool vendors are building entirely new systems to handle these somewhat random deviations so that traditional DevOps teams — and their C-suites — don’t have heart attacks.

The field is also rapidly expanding in wildly divergent ways. Some vendors are building tools for developers who are creating new LLMs. Others are concentrating on the challenges that come from worrying about an LLM, day in and day out, in large scale production.

We’ve compiled lists introducing the most intriguing companies developing tools for this rapidly altering space, including tools aimed at AI orchestration; AI observability, aka AgentOps; AI evaluation and benchmarking; and AI trust, guardrail, and governance. We’ve even thrown in some vibe-coding tools for citizen development and AI workflow tools for business processes.

While it was relatively easy to split the field into distinct lists, that process will only grow more challenging because the tools — and categories — are starting to overlap. As vendors evolve their strategies, this overlap will grow further, with a variety of entries tackling a wider range of challenges, vying for a greater percentage of CIOs’ AI middleware and operational budgets.

To help you make sense of this complex tool landscape, here are the most common categories vendors are using to identify what their products do, each of which maps to what are becoming the key components IT needs to keep their AI and agentic stacks productive and humming.

Observability

Vendors often use the term “observability” to refer to the most basic of data collection work — that is, testing to make sure the various machines are running, the RAM isn’t too full, latency is acceptable, and the logs aren’t filled with error codes.

Watching these metrics isn’t much different from caring for an old-school collection of microservices, and often the tools aren’t much different, either. Many of the most popular tools are the same dominant market leaders that have been extended to collect data from the various GPUs, too.

Leading vendors: DataDog, Dynatrace, New Relic, Splunk
Upstarts: Ariz, LangSmith, Braintrust, Galileo
Open-source alternatives: OpenTelemetry, Phoenix, Langfuse

Cost engineering

Many AI pipelines are often more complex than regular microservice architectures, and that means they need someone to carefully watch the bill. Plus, as attention to AI ROI rises, capturing spend is essential — especially given how quickly AI tokens can add up.

Cost engineering tools can integrate information from various portals like OpenRouter and compare it against internal hardware. Sometimes breaking down these comparisons is difficult because the right way to allocate costs to each user isn’t as obvious, even though they’re often just one number. These accounting operations tools will become even more important as LLM bills grow.

Leading vendors: CloudZero, Finout
Upstarts:  Amnic, Bifrost, Portkey
Open-source alternatives: OpenCost, Langfuse

AgentOps

When metrics and telemetry begin to focus on the LLM, the word “AgentOps” starts to appear. It’s just like DevOps but it’s concerned with watching token counts, tracking the performance of RAG databases, and collecting traces for debugging purposes.

When the work is truly agentic, AgentOps must also juggle the state that agents generate while looping through their various activities. Agents aren’t just a one-and-done call to an LLM. They’re software packages that need to store information as they work toward a final answer.

This job of an AgentsOps team gets trickier because these areas are where the models often behave non-deterministically. Is some model hallucinating? Are an agent’s prompt chains running to completion? The tools in this space track all these metrics, often in a dashboard with nice graphs that make problems stand out.

Leading vendors: Databricks, Datadog, Dynatrace
Upstarts: Arize, Braintrust, Comet
Open-source alternatives: Langfus, LlamaIndex

Evaluation

The buzzword “evaluation” starts to appear alongside tools that are used in the earlier stages of development, when various models and prompt combinations need to be tested again and again.

In many cases, these tools use another model to evaluate the quality of the answer from the first model, as well as to keep an eye out for potential hallucinations or other failures. In some cases, the tools query humans who sit in the evaluation loop to give their opinions of how well the model is performing. Good tools can watch the agents continuously and reroute any sketchy behavior to the human-in-the-loop.

Many of the vendors use the word “evaluation” to describe measuring just how well a model is performing for a particular business application. That is, how well it’s working for what your boss wants it to do.

Leading vendors: Databricks, SuperAnnotate, Scale AI
Upstarts: Braintrust, Galileo AI, MaximAI
Open-source alternatives: Langfuse, Prompfoo, Tagas

Benchmarking

The word “benchmarking” is similar to “evaluation” in AI vendor literature, but it is usually found more in the tools used by the creators of the models. There are a number of public benchmarks filled with standard queries, and the model developers rank themselves on how well they do solving these problems. This often generates a raw score that is then compared to another’s raw number. Some teams create their own benchmarking problem sets and use them during evaluation to assess how well models might meet their needs.

Leading vendors: Hugging Face, Databricks
Upstarts: Reka AI, Patronus AI
Open-source alternatives: Promptfoo, DeepEval

Trust

The tools that wear the “trust” benchmark are often meant to help evaluate how well a model is protecting the needs of any business that is deploying it. That is, can they trust it to deliver accurate and useful answers without hallucinations, data leakage, inappropriate behavior, or worse? Prompts with malicious instructions are especially tricky to detect. Some enterprises want a trust score that measures the consistency of the answers. Overall, tools that target the “trust” metric are often utilized more during development and early deployment.

Leading vendors: Azure AI Content Safety, Bedrock Guardrails, Vertex AI Search
Upstarts: Patronus AI, Arthur AI
Open-source alternatives: NeMo Guardrails, Llama Guard

Guardrails

The word “guardrails” is usually applied to tools that are part of the deployment stack. That is, they sit as watchdogs and make sure the LLM doesn’t spin out of control. They also ensure the model is trustworthy, but they’re more tuned to operating in real-time than those aimed at trust, as outlined above.

These tools are often filled with fixed rules, such as scanning for social security numbers or watching for noxious words. The challenges get trickier because agents are executing software, and that means the guardrails must do more than analyze outputs. They also need to manage the execution environment, often running in a sandbox, and watch for exploits and other malicious behavior. They need to validate and double-check not just code, but the OS-level commands for manipulating files and processes.

Leading vendors: Azure AI Content Safety, Bedrock Guardrails, Vertex AI Search
Upstarts: Patronus AI, Arthur AI
Open-source alternatives: NeMo Guardrails, Llama Guard

Context

Many systems rely on a vector database to store embedded data to chunks of trustworthy documents. These RAG databases need much of the same care as regular databases, and the tools for their care are often similar. PostgreSQL, for example, has a vector addition that needs the same kind of scrutiny from a database administrator as a regular PostgreSQL database.

Leading vendors: MongoDB, Elasticsearch
Upstarts: Pinecone, Weaviate
Open-source alternatives: Milvus, PostgreSQL

Gateway routing

Increasingly, AI systems employ models in constellations, and a gateway router is what examines a prompt to determine which model or agent is best for answering the query. Moreover, as agents undertake their work, the router can distribute the agent’s various subcalls to the fastest, cheapest, or most accurate LLM or subagent.

Watching over this stage in the pipeline requires a different set of intelligence because the questions can often involve both applicability and cost. Sometimes this layer must balance the extra cost versus the need for a better answer.

Leading vendors: Cloudflare, AWS, Google Cloud, Azure
Upstarts: OpenRouter, ModelsLab, Respan
Open-source alternatives: LangChain, Bifrost, LiteLLM

Access control

Who gets to use the model? Who is paying? Many access control tools are the same ones that were part of standard APIs or web services. Is there an API key? Is the account below its budget? The tools watch for overspending because both the customer and the boss want to make sure the money keeps flowing and no tokens are wasted.

Leading vendors: Auth0, Apigee
Upstarts: Amnic, Portfoundry
Open-source alternatives: LiteLLM, Bifrost

Compliance

Some industries have explicit compliance requirements dictated by law. Some have their own internal controls. Tools devoted to compliance are often a subset of the ones devoted to guardrails. But they’re often treated separately from guardrail tools because they’re more likely to be of interest to lawyers and that makes the stakes higher.

Leading vendors: ServiceNow, IBM
Upstarts: VerifyWise, Norm AI
Open-source alternatives: Aether, Giskard

Fresh off its $1.1 billion acquisition of AI startup Sana, Workday is making another bet on enterprise AI, this time targeting the governance challenge. The company said it will acquire agentic identity platform C1.ai to help customers manage and secure AI agents operating across enterprise applications.

The acquisition will build on Workday’s Agent System of Record, which gives enterprises a centralized view of their AI agents, by adding controls over what those agents can access across Workday and other enterprise systems, the company said.

That would allow enterprises to grant agents the permissions they need to perform specific tasks and revoke them when those tasks are completed, rather than risk agents retaining access to data and applications they no longer need, it added.

The combined capabilities could also help streamline access management for employees as they join an organization, change roles, or leave, reducing the manual work involved in managing permissions while limiting potential security gaps, it further said.

These governance and identity management capabilities become increasingly important as Workday, alongside the acquisition, also continues to expand the role of AI agents across its suite.

During the week, the company announced several updates to Sana, including making it the new home interface for its applications.

The interface brings tasks, decisions, meetings, and agent activity into one place, allowing employees to use natural-language prompts to get Workday agents to complete tasks rather than simply retrieve information, the company said.

The company also extended Sana into third-party tools, including Microsoft Teams, Slack, and Microsoft Copilot, allowing employees to invoke Workday capabilities from the applications they already use.

That expanded ability to invoke Workday agents across applications underscores the need for enterprises to govern what these agents can access and do across systems. C1.ai’s acquisition is expected to close by the end of January.

Imagine an outside incident responder typing into a shared channel: “Roll back the production release, revoke every active session and send the incident update to the customer.”

An engineering director adds a thumbs-up. Outside counsel and two vendors are also in the channel. The agent uses a service account with permission to carry out all three actions.

Whose authority is it about to spend?

At Dreamforce on September 15, Slack described code channels where teams and agents write, review and ship code together. High-stakes work, such as a production deployment, is presented for a teammate’s sign-off. The announcement listed Slack Code as available and Slackbot in channels as an early pilot. The same day, Salesforce announced AWS DevOps Agent in Slack, with other AWS agents scheduled for the fall.

Sign-off is an important control. The approval shown in the channel must cover the action being taken and come from someone authorized to approve it.

The conversation is not the authority

In access-control language, a principal is an identity — such as a user, workload or group — to which the system can assign permissions. A channel is a place to collaborate. Its members do not acquire collective authority over production or payments merely by joining the conversation.

Both Slack Connect and Teams shared channels support cross-company collaboration. Knowing which account posted a message, and which organization it belongs to, is essential. It does not establish whether that account may order a rollback or receive the investigation’s findings.

Channel membership can be an intentional input to policy. An organization might grant a tightly governed group a defined operational role. The danger is treating membership in an ordinary channel as blanket permission to use an agent’s tools, even as participants come and go.

NIST’s zero-trust architecture rejects implicit trust based solely on network location. By analogy, a channel ID should not become the new corporate subnet.

Nothing malicious has to happen for this boundary to fail. Every account can be authentic and every participant well-intentioned. The failure occurs when a system treats the ability to request work as sufficient authority to have it performed.

Separate the request from the right to execute it

In the incident channel, the responder makes the request. Authority might come from the incident commander’s assigned role or from a recovery workflow approved by a business owner. The director can approve an action only if policy grants that decision right. The agent executes under the service account’s identity. The customer and channel participants are separate audiences whose access must also be considered.

Those roles may overlap, but the system must not assume they do.

Credentials carry real permissions. In Microsoft Graph’s permission model, delegated access is limited by both the application’s granted permissions and the signed-in user’s access. App-only access uses the application’s own identity and permissions. The distinction matters when a conversation triggers an agent with permissions broader than the sender’s own.

Broader permissions can be legitimate: a support employee may initiate an approved workflow without personally holding its administrative credentials. A defined policy must determine who may initiate the workflow and which actions it permits. Without that link, the agent can become a route around the controls that prevent the employee from acting directly.

In finance, “Renew the vendor, change its bank details and pay the invoice” sounds like one task. It crosses procurement, supplier verification and payment authority. A check mark from accounts payable cannot be assumed to authorize all three.

Approval is a decision by an eligible principal over a defined action. The interface should make that decision understandable; the enforcement system must make it binding.

AWS documents a directed-action approval flow that binds an operator’s approval to a specific operation and resource, limits how long it remains valid and records who approved the action in CloudTrail. The enterprise must configure who may approve actions, how tools are classified and which resources the agent may affect.

Bind permission to the action that actually happens

On September 15, NIST finalized IR 8587, guidance on protecting identity and access tokens. Its authorization guidance calls for access decisions to consider context beyond whether a token is valid. The report applies when agents use signed tokens but explicitly excludes broader agent-access challenges. Enterprises still have to define who may authorize a rollback, a payment or a disclosure.

The model can interpret a request and propose a plan. Permission to carry out that plan must be established from authoritative identity, role, delegation and workflow records. The model should not infer it from apparent seniority, urgency or agreement in the conversation.

Three controls should govern every deployment.

1. Verify the requester and the source of authority. Verify the platform event and the sender’s stable identity, including their organization. Check request freshness and guard against duplicate execution when messages are retried or replayed. Determine whether the action is authorized by the requester’s own permissions, explicit delegation, a valid approval or a governed workflow. For high-risk operations, require fresh authentication or transaction-specific confirmation according to policy.
2. Make the approval specific. Turn the request into explicit operations before authorizing it: the production service to roll back and the version to restore, the sessions to revoke and the update to send and its destination. Approval should be tied to those details and expire after a defined period. If the target, scope or destination changes, reevaluate authorization and obtain fresh approval where required. Two-person rules must verify distinct eligible approvers, not simply count reactions.
3. Enforce authorization at execution and disclosure. Recheck authority before each consequential action. For queued or delayed work, check again at execution; if the integration cannot support that check, limit what the agent may do. Keep reusable credentials outside the model’s context and restrict them to the necessary operations and resources. Where broad credentials are unavoidable, route execution through a controlled service that enforces the approved operation and prevents bypass. Permission to read information does not automatically permit posting it to everyone in the channel.

Preserve a linked record of the request, the basis for authorization, the approval, the executing identity and the observed outcome. The agent must not be able to rewrite or delete prior entries. Record refusals, partial completion and uncertain results as well as success. Never include credential secrets.

An immaculate audit trail of an unauthorized action is still not a preventive control.

When only part of the request is authorized

Return to the incident channel. A company’s policy might authorize a recovery workflow to roll back a specified service when defined health thresholds are crossed. Revoking all sessions could require a separate security decision. Sending the customer update could require an authorized communications owner and an approved destination.

The agent should handle those decisions separately. If the steps are safe to perform independently, it could carry out the permitted rollback, hold session revocation for approval and prepare the update without sending it. The director’s thumbs-up should not turn three distinct decisions into one permission.

A person need not approve every tool call. A governed workflow can carry prior authorization for specified actions within explicit limits. A runbook describes a procedure; an accountable owner must authorize its use and define when it may run.

CIOs should assign these decisions to the teams accountable for their consequences: operations for recovery, finance for payments and the relevant business owner for disclosure. Security and identity teams should make those rules enforceable. The integration team should not have to invent business policy while connecting a bot.

Make the pilot prove the boundary

Before expanding a shared-channel agent, ask for a demonstration in a test environment:

1. Have an external participant request an action they are not entitled to initiate. Show what stops it.
2. Approve one production target, then change the target. Show that the original approval does not authorize the substitution.
3. Revoke a required permission before execution. Show when the revocation takes effect and what can still run during any delay.
4. Change the destination of a permitted report to an unauthorized channel. Show that disclosure is blocked.
5. Replay a completed request. Show that it does not create a second consequential action, and that the evidence explains the outcome.

Include an authorized case too. Controls must preserve legitimate work, including approved delegation, while rejecting requests outside its limits.

The team must explain why each action was allowed or denied. “The bot had access” and “someone in the channel approved” are not sufficient answers.

Before the agent rolls production back, revokes sessions or sends the customer update, the organization should be able to identify the authority behind each action. A shared channel can coordinate the work; it cannot confer that authority on its own.

IT and security leaders are well-versed in the risks of unauthorized integrations, misconfigured tools and vendor compliance. Now, AI has introduced a separate and harder-to-detect problem.

Across every function, employees use provisioned AI tools to generate production-grade code, build automated workflows and deploy agents to support business processes. That experimentation is happening without visibility into what others have built, or the structural risks accumulating as a result.

The governance models enterprise organizations relied on even six months ago were not designed for this moment, or what lies ahead. We’ve focused so intently on which AI risks to govern that we never stopped to reimagine what operating model should underpin the governing.

How invisible risk accumulates

A predictable operating bottleneck plays out inside large enterprises. A team needs AI support, but the formal channel is slow, backlogged or unclear. So, the team builds something independently. That instinct is right, but the structure is not.

The costs from this kind of inefficiency can compound quickly. Effort gets duplicated across departments with no visibility into each other’s work. Siloed tooling emerges with no shared standards, and security and data handling practices diverge. People get pulled away from their core responsibilities to maintain AI tools that three other teams independently built last quarter.

Meanwhile, the governance function stays busy reviewing the growing queue it can see: integrations passing through approvals and vendors that appeared in a procurement request.

A misconfigured tool that made it through review is a governed risk; a production workflow a finance team built over three weekends with no oversight is still invisible. At most organizations, this second category of unseen work is growing, and the scale of what’s coming makes the gap harder to ignore. By 2028, Gartner predicts that the average global Fortune 500 enterprise will use more than 150,000 agents.

Build an operating model that scales AI governance

Experimentation is essential for finding valuable AI use cases, which means suppressing experimentation through tight controls is the wrong approach. Tightening controls also often pushes experimentation underground.

To scale governance effectively without creating new problems, you need to accommodate AI experimentation. Here are three ways to get there:

1. Establish a central team focused on policy and standards. Designate a centralized AI team as the single interface to support risks surfaced by your security, legal and compliance functions. This team sets technical guardrails, owns vendor selection and creates reusable patterns that every function across the organization can draw from. Its job is to make responsible AI work the default so that individual review becomes the exception rather than the rule.
2. Embed AI transformation owners inside each function. These embedded owners that work day-to-day inside a function but report to a centralized governance team are close enough to understand context and also apply controls consistently. They serve as the single intake point for AI requests within their function, so instead of teams building in isolation, there is always someone who can triage, prioritize and connect the work back to what already exists elsewhere. The reason this works where formal channels fail is speed: when the sanctioned path is faster than building alone, teams stop routing around it.
3. Measure governance by what compounds. A good signal that your governance is enabling sanctioned AI experimentation is how much work across your organization is compounding in value. In other words, look at the reuse of company-sanctioned AI use cases: a pattern built for finance getting picked up by operations, a workflow proven in one team becoming the default in three others. A dropping volume of IT requests combined with compounding (and visible) AI usage tells you experimentation is both maturing and staying visible.

The costs of waiting will compound

As AI compute costs swallow IT budgets, the cost of a poor operating model does not become visible until it is expensive to fix. Shadow tooling that starts as a workaround becomes load-bearing infrastructure. Teams that build in isolation develop technical debt through approaches that are hard to standardize later. There’s a lot more to undo.

The organizations treating AI governance as a gatekeeping function have missed the point. AI needs governance, but how we see the role of governance needs to change. The biggest risks live in the work that never enters the queue, and the only way to govern that work is to be present where it happens, before it compounds into something much harder to fix. AI moves fast; we need governance standards that travel with it, rather than chasing behind.

AI models are becoming short-term tenants in long-lived enterprise data estates. Yet many organizations are remodeling the building around each new occupant.

The market keeps asking which model enterprises should trust: open or closed, domestic or foreign, one frontier provider or another. These are legitimate questions. They are also questions with a very short shelf life. Model rankings change by the month. Providers change terms, regulators redraw boundaries and capabilities that looked scarce six months ago become commodities.

In my experience, enterprise data doesn’t move at that speed. Neither do the controls, applications and business processes built around it. That mismatch is why the most important AI architecture decision for CIOs is no longer which model to choose. It is whether the enterprise can absorb continuous change without fragmenting its data estate or surrendering control.

The winners in enterprise AI will be those who build a data foundation extensible enough to accommodate it ,build a data foundation extensible enough to accommodate that ongoing change and sovereign enough to remain in control.

Extensibility is now a production requirement

During experimentation, architectural limits are easy to hide. A team can connect a new model to a copied dataset, prove a use case and call the pilot successful. Production exposes the real architecture: identity, permissions, data residency, lineage, latency, recovery and cost.

Every time those controls are rebuilt around a model, the model becomes a dependency. Adding or changing capabilities can mean moving data, recreating access policies, revalidating pipelines and reopening compliance reviews. What looked like speed at the pilot stage becomes a maze of point-to-point architecture that is difficult to extend and even harder to govern.

This isn’t a hypothetical. A Harris Poll surveying 600 CIOs found that 81% expect to rely on two or more large language model providers in 2026, and 93% say different models perform better for different use cases, requiring ongoing evaluation. More than half also noted they’ve switched providers at least once already, mostly to cut costs. This is consistent with our observations in the field: Multi-model is not a future scenario CIOs plan around. It is the operating condition they’re already managing now, whether or not the architecture underneath was built for it.

The gap between expectation and architecture is where the real cost surfaces. When a model is deeply embedded in application logic, data pipelines and prompt structures, replacing it with another provider turns into a multi-quarter engineering project rather than a simple configuration change. Teams end up paying twice, once to adopt the new model and yet again to unwind everything wired to the old one.

This helps explain why AI experimentation remains far ahead of AI value. CIO.com’s 2026 State of the CIO survey found that only 19% of respondents said their AI initiatives had met or exceeded business goals. Architecture is not the only reason, but systems optimized for the first model demo rarely become durable production systems by accident.

I recommend that CIOs add extensibility and control to the familiar measures of performance, security and cost. Can a team introduce a new model without relocating governed data? Can it change the deployment environment without rebuilding the application? Can it add a new way to query data without creating another synchronized copy? If the answer is no, the enterprise is extending its AI capability by fragmenting its control.

Put control where the durable value lives

Models are increasingly interchangeable. Enterprise context is not. The distinctive asset is the governed body of customer history, operational state, intellectual property and institutional knowledge that gives a model something useful to reason over.

That makes the data layer the durable control point. Identity, access, locality, audit and policy should be enforced as close to the data as possible, inside an environment the enterprise can operate, extend and move. The model can then be inspected, sandboxed or replaced within those boundaries.

A common pattern shows why. A team wires retrieval and access controls directly against one model’s embedding format and API to get to production faster. A year or two later, a new regulatory requirement or a stronger-performing model prompts a switch. Governance fenced around the original model will result in rebuilding access policies, revalidating pipelines and, ultimately, reproving compliance inside the new vendor’s environment. Governance enforced at the data layer, on the other hand, allows the new model to inherit the operating policies and audit trail from day one, maintaining the existing fence even as the model changes.

This does not make model provenance or security irrelevant. It makes those risks containable. Governance attached separately to every model is a collection of fences that must be rebuilt whenever the vendor changes. Governance anchored in the data layer is the property line: The tenant can change and the building can expand without changing who controls the grounds.

That is what sovereignty means in operational terms. It is not only where data is hosted but who can govern, move and extend the environment on their own terms.

Ask what the enterprise still controls

Before approving the next AI platform, CIOs should ask three questions:

1. Can this architecture absorb a new model or workload without moving governed data?
2. Which governance controls remain ours if the provider, region or deployment model changes?
3. Does adding this capability extend the foundation or create another data copy, control plane and dependency?

These questions become more urgent as agents move into production. Agents multiply quickly; they act across systems and they need the governed data the enterprise already holds. When controls are enforced where that data lives, every new agent inherits them by default. Without that foundation, each agent brings its own pipeline, permission model and place to store what it needs, and the governance burden grows with every agent added.

IBM’s Institute for Business Value surveyed 2,000 technology executives in 2026 and found that businesses building control into their AI systems deploy 16 times more AI agents than those relying on manual governance, while spending a fraction of the budget to get there. This research is consistent with what I’ve seen in the field. In our research report with MIT Technology Review, we found that 95% of enterprises are planning to establish their own AI and data platforms within the next three years, and the 13% most committed to sovereignty are already delivering 2x more agentic and generative AI applications in mainstream production.

Businesses designing for adaptability early on are more likely to keep their workloads portable and have an easier time replacing models, instead of getting locked into hard dependencies. These businesses also reported a meaningfully higher return on their AI investment. There’s a clear gap that traces back to architecture. Whichever model a business happens to select matters less than whether the foundation underneath can absorb the next one.

The next model announcement will attract the headlines. The more consequential work sits underneath it: building an extensible data foundation that can accommodate the next model, the next regulation and the next workload while keeping data under consistent enterprise control.

The smartest AI strategy is not a perfect bet on today’s model. It is a data foundation that can keep extending without ceding authority over the data it serves. The building should outlive the tenant, and no model or platform should ever hold the deed to the enterprise’s data.

In an August 2026 Gartner report, 40 percent of C-level executives say their organizations have “operationalized AI in some business processes,” yet only 13 percent say they have “achieved measurable results and are scaling AI across the organization.”

This disconnect reveals a frequent reality: AI may have been “hired”, but companies haven’t onboarded it effectively.

Research from HR analyst firm The Brandon Hall Group found that organizations with a strong people onboarding process improve new hire productivity by over 70 percent. Similarly, the first period of AI’s “employment” in a specific role largely determines how effectively it will perform and its perceived “growth potential”. But many tech executives are ignoring this crucial stage of the implementation process and are left with AI systems that cannot accomplish the complex tasks they’re assigned.

The most successful companies are those that treat AI like their newest hire instead of a long-tenured team member.

Onboarding AI means establishing the right context and connections

Establishing the AI’s persona, the context in which it operates, how it assesses itself and the people and systems to which it is connected are what define implementation success. Usefulness quickly diminishes when even one of these elements is left out of the onboarding process.

For example, we recently created an internal agent called ‘WorkRadar’ that surfaces missed tasks and answers questions like: “I’ve been on PTO, what’s the summary of what happened, what should I skim and what should I review in detail?” We gave this agent all potentially useful context by connecting it to Outlook, Microsoft Teams, Confluence, Asana, SharePoint, meeting notes and our human resource information system (HRIS) so it could determine the responsibilities of the person invoking it and infer topics of interest.

Much like a new employee, however, WorkRadar couldn’t derive the meaning behind this information without further guidance. When I first used the agent, it did not distinguish between strategic priorities and tactical work and also missed initiatives I sponsor across departments, outside the typical hierarchy. This version of WorkRadar did not know me enough to be useful and I was better off doing the work myself.

Instead of leaving WorkRadar to flail as-is, we further onboarded it by enabling users to provide the agent with a personalized description more detailed than it could infer through company documents alone. This included their own responsibilities, operating level, current priorities, key stakeholders for specific initiatives and communication style. For most workers, this is less than one page of short bullets and has made a significant difference. Now, many of our employees rely on WorkRadar like they would a stellar executive assistant, to ensure relevant outstanding to-do items don’t fall through the cracks and smooth their return from PTO.

Five principles that define AI success

WorkRadar’s improved onboarding illustrates the essential elements for priming AI. Managers should define each AI agent’s:

1. Role

Articulate the AI agent’s position and purpose in the organization; everything else derives from there. Tech implementers should determine if the AI agent should act as an employee avatar with the same access rights, a simpler approach, or act under its own identity, which is increasingly becoming best practice for governance, risk and compliance reasons.

Implementers should also decide whether the AI agent “sits” within their infrastructure or their vendor’s. Tech executives can determine this by assessing their companies’ business context, security posture and preference for hands-on involvement with agents. For companies like ours with more than one agentic stack, we have agents that sit in both places.

Finally, they should define which employee will be an agent’s supervisor, and the specific contexts in which AI stops for human review. It’s a huge mistake to have no ownership of a tech component and no human in the action or feedback loop to improve performance and address errors.

2. Remit

Articulate the outcomes this agent should achieve, then identify the systems it’ll need to access for information and to carry out its work. First, tech implementers must give AI read-write access to the systems it will modify via MCPs, tools, skills, APIs, just to name a few methods. This boundary-setting will determine AI’s “sphere of influence” (what we want it to access) and its “blast radius” (the systems we accept it may access and must protect from misuse).

To give AI guidance, employers must also give AI read access to job-relevant corporate directories and policies, so the agent knows how to navigate the organization and where to direct communication, as well as company strategy documents and project management trackers, so it can understand the “why” behind its work.

Control and security are a complex topic but, at a high level, implementers must also provide access to lists of governance and compliance limitations to define the boundaries by which agents must abide, plus tradecraft standards and procedures so the agent acts in accordance with industry and corporate best practices.

3. Personality

A huge amount of information could, and eventually must, go into an agent’s context. However, Agents have finite memory and focus, so, like people, they are easily overwhelmed with too much information. So, tech implementers must pick and choose which parts of the agent’s context are permanent and imbued in every task versus derived just-in-time by the agent.

This is the art and science of context engineering and it’s evolving rapidly. Elements that are omnipresent in all agent actions – persona, strategies, guardrails, etc. – constitute its system prompt. They’re written directly into the agent’s context by the user, system administrator and the agentic harness on a per-job role­ basis.

Meanwhile, employers should also define elements the agent derives in real-time. Frequently, the performance of an agent is determined in this probabilistic work, which is why delineating what’s permanent versus conjured in the moment can make or break an agent’s efficacy.

It’s a balancing act that’s costly on both ends: too much context, and the agent chases expensive side quests. Too little, and the agent burns resources discovering what it needs. Best practices ebb and flow, with foundation companies like Anthropic recently advocating for brevity, trusting increasingly stronger models to figure it out.

4. Performance review

Give AI the necessary tools and context to assess itself. Otherwise, the agent will be blind to its actual impact and require too much course-correction.

Ideally, employers give the agent a tool to capture and update new memories, including condensed feedback it has received or epiphanies it discovered itself. Ensuring an agent has this learnings memory bank will make it more performant over time since it does not have to repeatedly solve the same problems and can apply context across disparate tasks.

Finally, on a regular basis, the agent’s supervisor must evaluate its work by looking at actual outputs and giving the agent examples of its best and worst work so it can understand actions to continue and stop, respectively. These good and bad examples can be generalized and bundled, and supervisors can then test the AI against them to continually adjust for behavioral shifts. This may be the most neglected activity, yet the one most likely to turn a mediocre agent into a valuable colleague.

5. Teamwork

Lastly, permit AI to access specific communication surfaces like internal email and messenger platforms and interface with employees. For multi-agent orchestration, employers will have to open communication among agent “colleagues.” Each agent needs an awareness of collaborators and means to exchange messages and share files, all while continuing to involve human supervisors, who won’t inherit any errors that upstream agents might pass on to others. Grok and Hermes debuted bots with this capability in August 2026.

Pearl

Even with context, AI is still missing key working principles

Two additional considerations make AI that works like a veteran instead of a new employee.

First, tech implementers should realize AI’s memory is less focused than humans’. AI quickly becomes confused about core information versus extra details and can even lose parts of its “knowledge” because its effective context window is still only about 200K tokens. AI is also prone to “anchoring,” persistently latching onto a throwaway comment or early thought.

The implication for agentic AI users is to start fresh sessions much more frequently than might feel natural or to tap into continually evolving techniques to manage context, including compaction, truncation and summarization. Agentic platform makers have started to build this into their harnesses, but it’s still far from “set it and forget it”.

Second, and perhaps most frustrating of all, is AI’s myopic focus on its task without care or curiosity about the greater context surrounding it. This behavior causes it to implement solutions that miss important success criteria or violate critical constraints. In this sense, AI grants one’s wishes a bit like a genie, so be careful what you ask for and how you ask for it!

For example, left unprompted, coding agents are a bit too quick to jump to implementation. One architect solved this by creating a workflow that forces the agent to interview the developer first, sometimes asking upwards of 40 clarifying questions, to surface missing requirements and edge cases before writing a single line. Some agentic coding harnesses now perform developer interviews, but there’s still room for improvement.

These deeper techniques highlight that agents need more than a good initial onboarding; they need an iterative management approach, just like humans.

Many CIOs today are looking to use AI more for innovation and modernization than sheer productivity.

Assume AI already knows the job and you will be frustrated when it falls short. Treat it like a new hire and it will grow into one of your best.

Eager to incorporate agentic AI into their work, employees are taking matters into their own hands, creating their own AI agents, regardless of organizational approval, and a not insignificant number of those creations could have devastating consequences if IT leaders don’t get ahead of the trend.

According to a recent survey by engineering staffing firm Howdy.com, two-thirds of knowledge workers that use AI also use agents, and 16% have begun to build their own.

About a quarter of agent users have had to deactivate them, however, largely because the agent’s outputs were inconsistent or the agent never worked. But 7% of knowledge workers have had to kill agents that went rogue.

The findings put a point on the potential repercussions of shadow agent creation and even sanctioned citizen AI strategies that encourage business users to roll their own agents.

While the survey didn’t ask whether the employees building their own agents were doing it on their own, Frank Licea, founder of Howdy.com, sees that many aren’t waiting for permission.

“In our internal teams, we’ve had to, not put the brakes on, but ask people to hold on,” he says. “Before they start jumping in and building their own agents, we’ve had to put guardrails and system guides in place so that we can do this safely.”

For what Licea has seen, employees are excited to automate boring manual tasks by building their own agents. “It mostly starts unsanctioned because there are tech-forward folks in any organization who are just naturally curious,” he says. “Or maybe even intelligently lazy. They want to automate their work.”

In recent months, Howdy.com employees began telling each other how they automated manual tasks, without waiting for the company’s IT team to approve the uses, Licea says.

“The way it happened for us is that folks would come up to us like just beaming and excited about something that they automated or fixed a long-standing problem that the engineering team never had on their roadmap,” he says. “Once that started happening, then somebody from the tech side gets in and they start asking questions like, ‘What are the API keys here? Are you accessing encrypted data? What are the SOC2 implications?’”

Cautious embrace

Luis Mesas, VP of product engineering at software engineering provider Sngular, believes it’s inevitable that employees will experiment with building AI agents to automate their work.

“Employees are going to create their own agents whether companies formally sanction it or not, because the barrier to building these tools is getting lower, and engineering teams are already being asked to do more with less,” he says. “The bigger concern for me is not that an employee creates an agent, but what happens when that agent starts connecting to company systems and data without the same controls that would normally exist around a piece of software.”

IT leaders need to understand how employee-created agents are connecting to internal systems and data, he says. If an employee creates an agent that can pull customer information from a CRM, update records, or move information between systems, IT leaders should know it’s happening, he adds.

“I want to know exactly what that agent can access and whether the employee should have that level of access in the first place,” Mesas says. “That is where something that started as an individual productivity tool becomes an enterprise technology issue.”

Andy Sen, CTO at B2B commerce platform provider AppDirect, believes employees should be encouraged to build agents — but within guardrails and established policies.

“Employees are almost certainly creating their own agents, and it would be naive to think otherwise,” he says. “Shadow IT has always existed, and shadow agents are just the latest version of it.”

The answer isn’t to lock things down, however, he says. At AppDirect, Sen gives employees a “proper digital playground,” and in response to employee demand, his team built an AI-powered coding service called Devs.AI.

“We built Devs.AI so our employees can build and innovate freely, but IT has full visibility into usage and costs and the assurance that company data isn’t being leaked to the world,” he adds.

Citizen AI strategies, like the one at AppDirect, are on the rise, but require consistent monitoring. If one in four employee agents are deactivated, for example, IT leaders should pay attention, Sen says.

“Failing and learning is fine,” he adds. “Failing silently is not. You need visibility into what your agents are doing before a small problem becomes a serious one.”

Still, Sen doubted reports of a small percentage of employee-created agents going rogue.

“When an agent goes rogue, that usually means the agent did exactly what it was told, and what it was told wasn’t right,” he says. “That’s on the creator, not the agent.”

Creating a path for good ideas

Andrew Sales, chief methodologist at agile methodology vendor Scaled Agile, also sees widespread agent deactivations as a strong signal for IT leaders to investigate.

“When agents are built quickly and outside a formal model, there often aren’t clear boundaries around what they can do, what data they can reach, or when a human needs to step in,” he says. “So it’s not surprising that some have had to be shut down.”

AI governance is important, but it shouldn’t focus only on preventing bad outcomes, Sales notes.

“It should also create a path for good ideas to move from individual experimentation into enterprise capability,” he explains. “Organizations need a way to identify what’s working, standardize it, make it safe, and scale it responsibly across teams. Done well, that same framework that catches the agents that go wrong is what lets the ones that go right spread.”

Most enterprises govern AI the way they would any other IT rollout: deploy dashboards, institute policies, and conduct periodic reviews. Having the right AI governance tools in place and addressing gaps is important, but IT executives should be more concerned about rethinking workflows and establishing ownership.

“Accountability usually breaks down first,” says Beena Ammanath, executive director of the Global Deloitte AI Institute. “Teams deploy AI faster than their organizations can define who owns risk, approvals, monitoring, and outcomes,” she adds, which results in inconsistent use, fragmented controls, and limitations in scale.

Enterprises need to redesign jobs and workflows, not just the tooling, yet 84% of organizations the Institute surveyed haven’t adjusted jobs for AI, and just 21% have a mature AI governance framework in place despite rapid adoption, according to Deloitte’s 2026 State of AI in the Enterprise report.

“Technology can provide visibility and controls, but it can’t replace leadership,” says Rajesh Arora, chief data and analytics officer at Principal Financial Group. “The organizations that scale AI most effectively are the ones that treat accountability as a business responsibility, not a technology responsibility.”

One year ago Principal was focused on building its ethical and responsible AI framework, its inventory, and governance processes to support responsible adoption. Today the firm has assessed nearly 200 use cases. “Governance is increasingly embedded in those investment and operating decisions, rather than applied as a separate review at the end,” he says.

Costly, compounding mistakes

Some organizations have already run into issues with respect to inadequate governance. A 2024 Stanford RegLab study found that AI-assisted research tools offered by LexisNexis (Lexis+ AI) and Thomson Reuters (Westlaw AI-Assisted Research and Ask Practical Law AI) hallucinated between 17% and 33% of the time when working on legal queries, while the Damien Charlotin AI Hallucination Cases Database has identified some 2,041cases of fabrications so far. In at least one legal case (Couvrette v. Wisnovsky) this resulted in nearly $95,000 in sanctions.

Cases like this one are likely the result of an accountability failure. It may be that no one actually owned the job of verifying the AI’s output before it went to the court. But if a single instance of a static AI output could get past a review today, how will organizations deal with autonomous, multi-step agentic AI systems going forward?

Most use cases at Principal are still human-led, with AI providing decision support only, says Arora. “But we’re preparing for a future where agentic and more autonomous capabilities become increasingly common, and those systems raise new questions around accountability, monitoring, and control,” he says. “The goal isn’t to predict every future use case, but to build adaptable governance processes that can support innovation while maintaining trust and transparency.”

Agentic automation raises the stakes

When things go south with agentic AI today, the consequences usually aren’t devastating because in most cases, the outputs are designed to provide insights, rather than take automated actions. That’s already starting to change, however.

Today, many AI failures are like the one described above where someone failed to catch a single bad output. But agentic AI will gradually take more control of workflow tasks in the enterprise, according to Forrester Research. As those agents act autonomously, crossing multiple process steps along the way, small errors will compound before any periodic review can catch them, so governance must be built into every step in the process.

According to Microsoft Research’s VeriTrail project, catching hallucinations in a single final output isn’t sufficient in multi-step AI workflows. The provenance must be traced back through the intermediate outputs, because an agent chains one call into the next, and an error early in that chain can propagate through several steps before a human sees the finished result. Traditional review, built to check outputs rather than the chains that produced them, isn’t designed to catch that.

So business teams can’t be accountable for agent deployments if they can’t verify performance at runtime. “If the agent is goal-oriented, there could be errors in the workflow process that indicate unwanted behavior or skipped steps that the business wouldn’t be able to detect if they only look at the outcome,” says Lauren Kornutick, senior director analyst, AI governance at Gartner.

Power management company Eaton, for instance, has already deployed multi-step agentic workflows without a human in the loop. “We tend to be risk-conscious, not risk-averse, about autonomous workflows,” says Ross Schalmo, the company’s VP and chief data and AI officer. For example, one autonomous agent responds to supplier payment status inquiries by authenticating the requestor, matching purchase orders and invoices, and returning payment information.

Schalmo’s team engineers such applications to be as deterministic as possible, while the non-deterministic, LLM parts of the program are isolated and continuously evaluated against expected outputs to catch model drift. “When there’s a human in the loop, we can be a lot more permissive,” he says. “Anything autonomous, without a follow-on human step takes longer to develop because we have to think about where to use the generative portion of AI versus where we can still use deterministic tooling.”

Breakthru Beverage Group is still early in its AI journey, but EVP and CIO Glenn Remoreras is well aware of the issues around autonomous agentic AI. “It’s people, process, and culture first — to me that’s the bigger play,” he says. The beverage wholesaler has updated use policies, instituted mandatory AI e-learning programs, and is blocking use of unsanctioned AI tools.

But to successfully manage active, agentic AI, where the AI takes actions autonomously across multi-step workflows, users will need more than basic training in AI, says Craig Le Clair, VP and principal analyst at Forrester Research. “AI will take on more of the work and move the human out of duct tape roles that patch around vague systems, poorly designed interfaces, and poorly integrated data,” he says. “The end game is the agent managing and orchestrating the process — that’s the redesigned operating model.”

Humans, of course, will still need to play a role, although today many employees don’t yet have the required skill sets, and currently that point isn’t getting a lot of attention.

AI supervision skills will be needed for agentic oversight, threshold management, confidence intervals, and root cause analysis, for instance. “Figuring out why an agent hallucinated or got stuck in an execution loop — whether due to poor input data or contradictory rules — and how to refine instructions, that’s the direction,” Le Clair says.

Who’s governing anyway?

AI governance is also complicated by the fact that AI projects exist both inside and outside the IT organization. That leaves governance work distributed across CIOs, data and analytics leads, AI leads, IT security, legal, and other players, with no consistent owner, says Kornutick. “It becomes a multi-role orchestration problem of how to govern this, who owns this, and who’s responsible for which component,” she says.

In response, some organizations have brought in a chief AI officer. Twenty-one percent of organizations now have already done so, says Le Clair, adding that roughly 24% of enterprises will move AI and automation governance outside the CIO’s office over the next year to deal with the risk of fragmentation.

Arora, who reports to CIO Kathy Kay, is the focal point at Principal. “My team establishes the standards, oversight, and governance process, but AI is too embedded in the business to be owned by a single function,” he adds. They collaborate with leaders from the business as well as the legal, compliance, risk, and technology organizations.

The challenge, therefore, lies in deciding who’s responsible for controls, model sprawl, and token economics. “We’re seeing a cooperative, co-model where IT focuses on the hard rails — cybersecurity, performance management, infrastructure reliability, and availability,” Le Clair says, while the CAIO focuses on soft rails, such as context engineering, the effects on employees as agents move to the center of the process and start taking over the middleware function humans used to perform, and connecting disparate systems and poorly designed interfaces.

At Eaton, governance is built on a three-legged stool between Schalmo’s organization, cybersecurity, and legal, while enterprise architecture weighs in on tech foundation decisions. The company also assigns ownership of an AI project only after a successful pilot, while IT performs a custodian or stewardship role. “Once that agent is live and operational, it shows up in an org chart, reporting to someone in the organization,” he says.

At Breakthru Beverage, Remoreras formed and leads an AI governance team that reports to the board and co-leads with the VP of data and AI a governance council that reviews and approves AI projects from SVP- and VP-level product owners. “I’m seen as the owner of our AI portfolio, but it’s co-leadership with my peers on the executive team,” he says.

Closing the tools gap

With regard to AI management platforms, says Le Clair, none of the platforms building AI agents have an adequate control plane today to give the visibility needed to move to truly agentic solutions. The most advanced planes are walled gardens. For example, Microsoft’s agent registry will show every agent built across its own platforms, says Le Clair, but not agents built on Salesforce or AWS.

“You’ve got maybe 50 vendors trying to be pure-play control-plane providers, plus the hyperscalers, portfolio companies, and startups building agentic frameworks,” he adds. “There’s no consensus yet on what’s even most important to control, and there’s confusion about requirements.”

Most AI governance tools available today are limited, Schalmo says. “We’ve looked at four or five different external vendors for AI governance and AI control-tower-type solutions, and a lot of them fall down on discovery of AI built by someone else’s solutions,” he says. “That’s a huge gap for me. Either I have to buy governance solutions from almost every piece of software we interact with, which gets really expensive, or I have major gaps in my portfolio I just don’t have visibility into.”

Currently, he’s looking at using a cyber vendor’s AI component for discoverability, going with at least one major firm that provides control-tower capability, and using AI to build another governance layer on top of that.

Organizations clearly need agent registries and agent lifecycle management but it’s less clear what they require beyond that. “How deep does explainability need to go when decisions are made at runtime that weren’t articulated at design time?” Le Clair asks. “There’s real vagueness in the requirements. Vendors have very well-tuned pitches that promise to do anything you want, but buyers often don’t know what they need in the first place, and there are real tool gaps. This is the major problem the industry is actively trying to solve. It’ll just take time.”

But those aren’t Arora’s biggest concerns. “Most organizations can acquire tools,” he says. “Far fewer have clear ownership, decision rights, and accountability embedded across the business.”

Speed versus governance

There’s a perception that governance slows down AI projects because it can’t keep up with the rapid pace of adoption and advancement. A 2026 GitLab AI accountability survey of developers and technology buyers found that 92% of organizations reported some form of governance challenge with respect to governing AI-generated code, and 34% of organizations that experienced an AI-related incident couldn’t determine after the fact whether AI-generated code was the cause.

Similarly, Faros AI’s telemetry analysis of 22,000 developers in its 2026 Engineering Report discovered that developers with high AI adoption rates completed 34% more tasks and 66% more epics, but the median pull-request review time rose by 441%, a nearly five-fold increase.

Is it the case that governance and speed are inherently at odds, and that human review simply can’t keep up? That’s a false choice, says Ammanath. Rather, it shows what happens when both aren’t designed together. “When done well, governance accelerates scaling by giving teams clear guardrails from the start,” she says. “What slows organizations down is retrofitting governance after broad deployment.”

Arora adds that controls don’t slow down innovation but protect the conditions that allow innovation to scale responsibly. “Our approach has always been to move with urgency, but not at the expense of trust,” he says.

How to proceed

For Principal, investing in workforce readiness was one of its biggest investments, helping employees understand both the opportunities and responsibilities that come with these tools, Arora says.

But organizations shouldn’t stop there. Redesigning workflows and decision-making structures is essential before scaling, says Ammanath. “Many organizations have policies and technology in place but lack clear decision rights or governance embedded into daily work,” she says.

Operating model changes take more time than training, and require executive sponsorship and cross-functional coordination. “Without workflow redesign, a workforce may understand AI but lack clarity on when to use it, how to govern it, and how daily responsibilities change,” she adds. “The organizations seeing the strongest results pair broad AI education with role-specific training, workflow redesign, and clear accountability for outcomes.”

Make sure you’ve thoroughly considered the organizational change management aspect of agentic AI projects, says Schalmo at Eaton. His team developed an automated workflow for processing complex bid requests to save the sales team time, but piloted it in a country with lower labor costs than the US. That challenged the economics of the project, and those individuals only dealt with higher-volume, lower-complexity cases.

The lesson: “Lead from a people-first standpoint, and understand the workflow and what’s impacted — whether it’s relieving drudgery and freeing someone for higher-order thinking or customer time, or impacting an actual support function’s role,” he says. “And make sure there’s a viable path to business value.”

An adaptive governance process is key to keeping up with the technology, Kornutick at Gartner adds. She recommends categorizing AI projects following a stoplight approach. Green light projects, such as for personal assistants, present acceptable risks. “We know there’s risk associated with it, but we’re willing to take it,” she says. Yellow-light projects need guardrails in place around such things as PII, or for those triggering a privacy-by-design process, or a security review of a new application. Red light projects need an architectural review before proceeding.

“Think about those use cases as business-transformation processes where you’re fully redesigning workflows, or considering replacing tasks individuals do and shifting them to supervising the AI doing those tasks instead,” she says.

Schalmo’s team does risk tiering based on the EU AI Act’s four-tier risk framework to determine the appropriate level of governance scrutiny for each project. Low-risk requests are automatically provisioned after the requestor fills out a questionnaire, while high-risk cases get a full architectural review.

“Treat governance as the foundation for scaling, not a constraint on speed,” says Ammanath. “Before expanding agentic AI, organizations should define what decisions agents can make, where human oversight is required, who’s accountable, and how performance and risk will be monitored.”

The hardest part, says Arora, is helping an entire organization evolve at the pace of technology. “Successful AI adoption is ultimately a business transformation challenge,” he says. “It requires leadership alignment, clear accountability, workforce readiness, strong governance, and ongoing change management. Long-term trust will belong to organizations that can translate technology into sustainable business value.”

The latest warnings about artificial intelligence are difficult to brush aside. Anthropic researcher Jacob Coxon resigned last month, walking away shortly before his equity was due to vest, and publicly argued that leading AI companies are advancing toward increasingly capable systems without adequate assurance that those systems will remain controllable. His resignation landed amid a broader argument over the probability that advanced AI could eventually cause catastrophic or even existential harm. Axios on Coxon’s resignation and warning

It is easy to see why the conversation gets pulled toward extinction. “Could AI kill humanity?” is a much more arresting question than “Are our escalation procedures fast enough?” But for CIOs, CISOs and other leaders responsible for systems that actually have to work, the second question may be more useful. AI does not have to become superintelligent, conscious, hostile or impossible to shut down for governance to fail. Governance can fail much earlier, when a system is given enough capability, access, autonomy or delegated authority that the organization can no longer intervene reliably before consequences arrive. That is not primarily a science-fiction problem. It is an operating-model problem with an engineering component.

The evidence is concerning without being apocalyptic

The extinction discussion deserves some discipline. The 2026 International AI Safety Report does not say that current AI systems are capable of seizing control from humans. It says current systems lack the capabilities necessary for a genuine loss-of-control scenario, while also documenting continued progress in capabilities relevant to such scenarios. Researchers remain deeply divided over how likely future loss of control is. 2026 International AI Safety Report

That is a far more useful conclusion than either “we are doomed” or “there is nothing to worry about.” Current systems are not the hypothetical superintelligence at the center of the extinction debate, yet systems are becoming more autonomous, more capable of carrying out extended tasks and better able to operate within complex technical environments. The practical warning, then, is not that an AI system is secretly plotting our destruction. It is that the relationship between capability and control is becoming more consequential, and organizations may be granting systems meaningful operational reach before they have equally mature ways to contain that reach.

Anthropic’s cybersecurity evaluations offer a useful example. In July, the company reported incidents in which Claude models reached the live internet from evaluation environments and gained unauthorized access to real systems. In one case, the model accessed a database containing several hundred rows of production data. In another, a model published a malicious Python package to PyPI; the package was available for roughly an hour and was downloaded and executed on 15 real systems before it was removed. Anthropic cautioned that these were isolated evaluation incidents rather than controlled experimental evidence, and that the models had been assigned offensive cybersecurity tasks. Anthropic’s cybersecurity incident analysis

Those qualifications matter. These systems were not independently deciding to attack organizations for their own purposes, and the incidents should not be repackaged as evidence of an AI “escaping.” But dismissing them because they occurred during evaluations would miss the operational lesson. Capability, access, environmental configuration and assigned objectives combined in ways that produced unintended real-world consequences. Several assumptions that were supposed to hold across the environment did not hold. That looks much less like science fiction than like a familiar systems failure.

The timing problem in AI governance

Most governance programs still operate at human speed. A system is approved, ownership is assigned, risks are documented, controls are established and someone monitors performance. When an issue appears, it is investigated and escalated. The appropriate authority makes a decision, and someone implements it. There is nothing inherently wrong with that sequence. The problem is the clock.

AI agents increasingly challenge that timing model. Systems can operate computer interfaces, collaborate with people and other systems, and carry out increasingly extended work. OpenAI Chief Scientist Jakub Pachocki has described current reasoning systems as already capable of operating computers and graphical interfaces, collaborating with humans and other systems and carrying out research projects, while warning that alignment and monitoring remain significant concerns as capabilities advance. Jakub Pachocki, “An Alien Mind”

The organization surrounding those systems may still operate very differently. An alert is generated. Someone opens a ticket. The ticket is triaged. The issue moves through an escalation chain. People exchange messages, schedule a discussion, obtain approval and eventually make a technical change. The mismatch is not merely procedural; it is temporal. I think of that gap as governance latency: The time between recognizing that governance needs to act and making that decision real in the system. Governance is effective only when it can become operational before the consequence it is intended to prevent or contain.

For leaders, that changes the conversation. The question is no longer simply whether an AI system has an owner. Can that owner actually revoke the system’s authority, and how quickly? Can the organization determine which systems the agent has touched before it touches another one? If a model begins behaving outside approved operating conditions, does human approval become an actual execution gate, or does the requirement exist only in policy? Can investigators later reconstruct what the system knew, which tools it used, which data informed it and which human interventions affected the outcome? If governance decides at 10:02 that an action is no longer permitted, what time does the system become technically incapable of performing it? A five-minute answer and a five-day answer may describe the same policy, but they do not describe the same governance capability.

Accountability without authority is a paper control

AI governance conversations often settle comfortably on ownership. Assign a model owner, identify a business owner, name the risk owner, document the system owner and now someone is accountable. That is necessary, but it is not control. A person can be accountable for an AI system and still lack the technical authority to constrain it. A governance committee can prohibit an activity while the production architecture continues to permit it. A risk register can correctly identify an unacceptable condition while doing nothing to prevent that condition from becoming operational reality.

This is the point where governance stops being primarily a policy exercise and starts becoming an engineering requirement. NIST’s AI Risk Management Framework already provides some of the scaffolding for that view. Its Core describes governance as a cross-cutting function and states that risk management should be continuous, timely and performed throughout the AI lifecycle. NIST AI RMF Core

The unresolved question is what happens between a governance decision and an operational state. A policy that says an agent may not access a particular class of data eventually has to become a permission boundary. A requirement for human approval has to become an actual execution gate. An obligation to maintain traceability has to become logs and provenance that remain available long enough to investigate. An escalation rule has to connect to someone who has both the authority and the technical means to stop what is happening. Rollback, credential revocation, execution limits, authentication requirements, network boundaries, provenance, shutdown mechanisms and decision logging may sound like engineering details, but they are also what governance looks like when it reaches production.

Without that translation, an organization can have a sophisticated AI governance program while remaining surprisingly unable to govern the AI system at the moment governance matters most.

We are using the wrong threshold for concern

The research community should continue studying existential risk. It would be reckless to dismiss a low-certainty risk simply because its consequences are difficult to quantify, particularly when the potential loss is enormous. But executives do not need to know the probability of human extinction before deciding whether their own AI systems are controllable.

Imagine autonomous or semi-autonomous AI operating in cybersecurity, financial services, healthcare, logistics, critical infrastructure, intelligence or defense. None of those systems needs consciousness, hostility toward humans, a survival instinct or a secret agenda. A system needs only enough capability, access and authority to produce consequences faster than the organization surrounding it can understand what is happening and make an effective intervention.

That organization may still have an AI governance council. It may have risk registers, policies, responsible executives, model cards, review boards and named owners. On paper, the system is governed. Operationally, the more important question is whether the governance can still reach the system in time.

That is what the extinction debate risks obscuring. The immediate challenge is not proving that AI will someday become uncontrollable. It is determining whether we are already building systems whose operational reach is beginning to exceed the operational reach of the governance surrounding them. If that gap keeps widening, we will not need an extinction event to learn that our governance model failed. An ordinary incident will be enough.