The true promise of agentic AI lies in agents maneuvering, making decisions, and taking actions autonomously throughout the enterprise, but few IT leaders appear ready to give bots widespread control.

According to a KPMG survey from July and August, 62% of IT leaders reported they were building, deploying, or developing AI agents, up from 53% in the previous quarter. But many IT leaders still say they are uncomfortable with the prospect of turning over internal processes to agents, with some exceptions for basic, low-risk functions like tier 1 IT support.

Jim Olsen, CTO of AI lifecycle management and governance vendor ModelOp, sees agent autonomy making inroads in limited circumstances — for example, to process customer refunds or credits without human intervention. But early lessons in the risks of full autonomy remain top of mind for IT leaders.

“We saw the large adoption of Clawdbot and OpenClaw by individuals where they would allow it full access to their email and gave it full autonomy,” he says. “For some people that worked out well, but it was a large risk exposure.”

For IT leaders deploying autonomous bots, Olsen recommends confining them to a strict sandbox with well-defined policies to limit their autonomy.

“I do not see allowing agents full and complete autonomy to do whatever they want being used in an enterprise in the near future,” he says. “But restricted autonomy is, in fact, already in use today.”

Activities that aren’t mission critical and don’t have access to sensitive information will be where IT leaders first grant agents full autonomy, Olsen predicts. “As trust improves, so will the breadth of use cases,” he adds.

Some IT leaders are concerned about “autonomy creep,” when agents quietly take more authority than they are given. And many AI and security experts warn that human-in-the-loop guardrails don’t always give employees the authority or time needed to fix agent errors.

Still, IT leaders have several methods for enforcing runtime policies and keeping agent behavior within acceptable goals, including API gateways, OS-level enforcement, and credential management, Olsen says.

“This means having clear policies in place and being able to tie them to the specific agent and use case,” he adds. “Some of these solutions are available now, but a more overall cohesive strategy is needed to bring about this minimal level of trust.”

Defining the sweet spot for autonomy

With about 2,000 agents deployed within her enterprise, Siroui Mushegian, CIO of cybersecurity vendor Barracuda Networks, still has human employees checking bot actions.

“We have many agents that are very close [to autonomous] — many of our ticketing automations are in the ‘trust-but-verify’ category with the agents that oversee them,” she says. “The more we see success, the greater comfort we have, but I am still not 100% certain I want to allow zero human accountability, especially when it comes to auditable processes.”

Mushegian expects autonomy to expand as agents demonstrate they can handle specific tasks reliably. “The consequences of a mistake should determine how much evidence you need before giving an agent more authority,” she says.

Autonomy will likely come to agents executing specific tasks within a function, she adds. Routine internal support requests are a reasonable candidate for autonomy when the answer is well documented, the action is limited, and a mistake is straightforward to correct, she suggests.

Less likely candidates for autonomy are agents that can change access permissions or make changes to a production system, Mushegian adds.

“Even within one workflow, an agent might complete some steps on its own and need approval for others,” she says. “I’d give it only the authority it needs for the task, with human approval for decisions where a mistake could have serious consequences.”

It’s about the action, not the agent

Joshua Rowe, co-founder and CTO of household bills app provider Taupia, agrees that IT leaders shouldn’t give agents complete freedom in every circumstance.

“I’d draw the line at the action rather than the agent,” he says. “The same agent can act alone on one step and need sign-off on the next.”

However, Rowe sees a lot of areas where agents can run near-autonomously right now, including read-heavy work such as gathering data, triage, drafting, and comparing options.

Taupia is testing agent functionality that compares energy deals and takes steps to allow customers to switch providers without its employees approving each step. However, the customer still would have to review the change and confirm it before it goes into effect.

For agents to get autonomous authority for irreversible actions, the timeframe doesn’t depend on AI models getting smarter, Rowe says.

“It depends on how quickly an organization can make each action reversible or bounded, with spending limits, cooling-off periods, rollback, and an audit trail good enough to undo a mistake,” he says. “Once an action has those, it can go autonomous. Where it can’t be undone, like signing a contract or moving money out of an account, I’d keep a person approving it for the foreseeable future.”

A question of deliberate design

Maciej Walaszczyk, CTO at AI customer service platform provider Text, points to customer service as an area where IT leaders have begun to cede control to agents.

“In customer service, which is my field of expertise, agents can indeed answer customer inquiries without the need for constant supervision, as they learn from knowledge provided by humans,” he says. “The scope of autonomy depends on permissions and the consequences of an action. Furthermore, the scope of these responses can be tailored to specific needs.”

But Walaszczyk says he would not hand entire departments over to agents. Instead, IT leaders should delegate specific processes where they can define the goal, set boundaries, and verify the outcomes, he recommends.

“An agent acting on my behalf cannot have more authority than I do,” he adds. “I don’t think it’s a question of being comfortable or even courageous, but of deliberately designing the environment, permissions, instructions, and oversight.”

Trust will come from real work and verified results, and humans must remain in the loop to check agent outcomes in many cases, Walaszczyk adds.

“Oversight must be real,” he says. “It takes time and attention, and it means much more than clicking approve. Saying ‘AI did it’ does not remove their responsibility. Irreversible actions or those with major consequences require human review.”

The businesspeople adopting AI notetakers are not being reckless. They are responding to a real business need. Our teams are in meetings all day with companies we support, and the idea that we could read a clean summary of every one of them at the end of the day makes us better partners to our customers. At the AI-first startups we spend time with, a new account manager can walk in on day one, inherit the full recorded history of every customer conversation, and ask the AI what that customer actually cares about. That is not surveillance culture for its own sake. It’s a better way to serve people.

Still, AI notetakers present cyber risks, as we have been warning about for the past year: shadow adoption measured in hundreds of accounts in a matter of weeks, vendors without SOC 2 or real encryption, consent liability, litigation exposure. In April, one of us stood on stage at the [un]prompted conference and argued that the most important attendee in your meetings is no longer a person. AI notetakers decide what matters, assigning the action items and providing the summary that becomes the meeting’s official record.

But our early warnings about AI notetakers shared an assumption we no longer hold: that the notetaker joins the meeting. That assumption quietly died this summer, as the notetaker has left the meeting platform and moved onto the body — and policies haven’t noticed let alone caught up to this change yet.

From the meeting to the wrist

On July 28, Granola, the AI notetaker we see most often in our worlds, launched an Apple Watch app. One tap on the wrist and it records whatever happens around you. No bot joining the call, no banner announcing itself, no laptop open on the table.

The same week, DEF CON banned Meta-style smart glasses from the conference floor over recording concerns. Meta, meanwhile, had already acquired Limitless, the AI pendant company, folding always-on personal recording into the largest consumer hardware pipeline in the world. One public company we’ve worked with now screens attendees for smart glasses before all-hands meetings where unreleased plans are discussed.

We’ve tested this ourselves, putting the watch under a jacket sleeve, tapping once, and recording an entire evening walk. Clean transcript. Nobody could have known.

In the United States, recording someone without their permission depends on which state you are standing in. California says you can’t, but many states say you can. More importantly we still have no norms of disclosure, and the hardware is now designed to be invisible.

The record is not neutral

Since the April talk, we have met a steady stream of people telling us how they game these systems. One person described doing prompt injection against a notetaker in a live meeting, right as we walked offstage. Another explained the simplest trick we have heard: join early, and for the five minutes you are alone with the notetakers, calmly narrate the outcomes you want captured as the most important takeaways. A third said he figured out how to silently crash other companies’ notetakers, so the meeting ends and there is no record. There is academic research behind all this. Call it manipulation or call it strategic communication. The methods work.

Notetaking systems also fail on their own. We have watched a transcript mishear one word, then reinterpret the rest of the conversation to fit the error. Colleagues with accents the models handle poorly carry more of that risk than the rest of us. And these tools are no longer stenographers. They have memories. Granola pulls context from prior meetings and background on the people in the meeting into today’s meeting notes. That is useful, but it also means a summary shared outside your company could leak reasoning from meetings the other party was never in.

Then there is the case that should be on every policy writer’s mind. Last November, a Campbell’s security executive was terminated for what they said during a meeting secretly recorded by an employee during what he thought was a private lunch conversation. The recording surfaced months later through an employment lawyer, and the executive was gone within days. Whatever you think of what was said, notice the mechanism: no notetaker bot, no meeting platform. One person, one device, one conversation the speaker believed was off the record. There is no such thing anymore.

What a serious policy looks like

Boards and legal teams are now asking their CISOs for notetaker policies. Having helped several organizations think through this, here is where we land.

Transparency always. If a recording tool is running, everyone present knows. The lightest version we recommend: Change your display name to “Name + AI notetaker.” And the rule must bind employees, too. An employee who records a colleague without disclosure is committing a fireable offense, full stop, before we ever discuss what the recording contains.

Classify your meetings. At one firm, we wrote a formal policy naming which meetings may never be recorded. Yours might include M&A, legal, HR, and board executive sessions. Decide deliberately, in writing, before the tools decide for you.

Govern other people’s notetakers. When an auditor walks into your building, their laptop has to meet your standards. Apply the same principle to recording. If another organization records your employees, you should know, and you should get a copy.

Choose your retention posture. A full transcript is a different corporate record than a summary. Decide whether transcripts are destroyed by default, with legal holds as the exception, or kept forever. Do not let the vendor’s default decide.

Treat notetaker vendors like what they are: startups holding your most sensitive conversations in their cloud. Vendor security review, data-reuse terms, indemnification language. Brewer v. Otter.ai, a class action alleging Otter’s notetaker recorded meetings without everyone’s consent and used the recordings to train models, and the sudden shutdown of Novacy both happened for a reason.

Assign a human. Every transcript that becomes a record needs a named owner responsible for reviewing its accuracy, no matter whose tool recorded the call.

Remember that some may train employees to steer the transcript, the way a lawyer protects the record in a deposition. We understand the instinct, and reviewing the record for accuracy is exactly right. An organization that quietly optimizes its official records has bigger problems coming.

We used to debate whether to let the notetaker into the meeting. That debate is over, because the meeting is now everywhere the recorder goes. Write your policies for the world where the microphone is already in the room, because it is.

Companies are moving fast to put AI into chatbots, virtual assistants, and other customer service tools. But for CIOs, that comes with a risk: if the technology gives customers the wrong information, makes biased recommendations, or mishandles data, it can immediately damage the trust companies have worked hard to build.

Unlike conventional software, gen AI doesn’t always give the same answer to the same question. An answer can sound confident and convincing while still wrong. And when customers rely on AI for information about their finances, insurance, or health, a mistake can have much greater consequences.

Sumit Johar, CIO at cloud-based software company BlackLine, understands that inaccurate AI responses hurt the customer experience, force companies to spend money and resources to fix problems, damage the brand, and create regulatory risks. AI systems can also expose confidential information, making it important for companies to understand the data sources behind their responses.

“An AI failure creates a massive trust issue around the core solution being provided,” Johar says. “In the financial sector, bad AI advice or outputs can compromise balance sheet accuracy and lead to severe legal and personal liabilities.”

For Citizen Health CTO Daniel Wang, for instance, the stakes are particularly high because the company’s users include rare disease patients and caregivers.

Citizen Health works to ensure that important information provided by its AI can be traced back to the original source, such as patient records, clinical information, or medical studies. The company also reminds patients to talk with their healthcare providers before making medical decisions.

Wang says one of his biggest concerns isn’t just that AI can make a mistake, but that a wrong answer can sound right.

“An AI system can present an answer that sounds caring, informative, and authoritative, even when the underlying information is inaccurate,” he says.

That risk has affected how Citizen Health releases AI products, and has delayed AI launches when the company needed more time to understand how the technology was coming up with its answers, or to make sure those answers were accurate. So instead of rolling out new AI tools to everyone at once, smaller groups of patients might be tested first, and their feedback used to make improvements before a wider launch.

Citizen Health also tests its AI tools before launch using questions with known answers, and uses both automated testing and human review, watches for unusual AI responses, limits what the AI can answer, and gives patients ways to provide feedback.

Moving at the speed of trust

Other tech leaders are taking a similarly cautious approach. Jonathan Tushman, CTO and CAIO at Hi Marley, an insurance technology company, says it’s had tech that was ready to ship for years but chose to spend more time on testing, observability, legal review, and training before releasing it.

“If it weren’t for that trust-and-governance bar, we’d have gone live a year earlier and had faster adoption,” Tushman says. “We chose not to.”

Hi Marley calls that approach working at the speed of trust. For Tushman, releasing customer-facing AI also requires the ability to see what the system is doing once it’s running. That includes monitoring individual conversations as well as looking for patterns across tens of thousands of interactions. Companies also need kill switches that can shut down a system, and the ability to roll back changes if something goes wrong.

But technology alone isn’t enough. Customer service, IT, DevOps, and CloudOps teams all need procedures to respond to AI problems, Tushman adds.

Udaya Bhaskar, senior application security analyst at global farming company Corteva Agriscience, says one mistake companies make is moving too quickly after seeing an impressive AI demonstration without fully testing how the tech will perform with actual customers.

He adds that companies can also run into problems when they rely too heavily on vendor claims, or launch AI tools without giving customers an easy way to reach a person when something goes wrong.

The hidden cost of AI mistakes

The pressure to deploy AI is often tied to the promise of greater efficiency. A chatbot that handles more customer questions, for example, can reduce employee workloads. But those time and cost savings can shrink quickly if employees have to spend their time checking and correcting the AI, something referred to as the hallucination tax.

Companies may budget for developing the technology without fully accounting for ongoing costs of running and monitoring it.

Jess Lampe, global lead technologist at Launch Consulting says those hidden costs can include employees spending time checking AI responses and redoing work when the technology gets something wrong, as well as regulatory and legal risks. Another cost can be harder to measure: loss of customer trust.

Chris Slovak, global field CTO at managed enterprise AI delivery platform Unframe, sees the resulting damage to a company’s reputation as one of the largest hidden costs.

“Anything customer-facing is a representation of your brand,” Slovak says. “Yes, poor implementations with bad experiences have a material cost from the perspective of tokens. However, the impact on your brand is far more costly. Often, companies add AI to deflect customer support. Consider you have a customer who’s already frustrated, and meeting AI makes it worse. The loss in trust can be massive.”

But Kevan Yalowitz, global software and platforms lead at Accenture, says mistakes are going to happen whether work is performed by people or technology.

“The true hidden cost is destroying user trust when mistakes happen, regardless of the source,” Yalowitz says. “The most successful organizations quickly admit when these mistakes happen, provide a transparent explanation, and explain how it was an opportunity to learn and ensure the mistake doesn’t happen again.”

Human oversight remains critical

For CIOs, preventing those problems means putting controls around AI before it reaches customers.

Johar says human oversight is particularly important for critical decisions. In highly regulated industries such as finance, AI shouldn’t independently handle decisions where accuracy is essential. Companies also need reliable data feeding their AI systems, and the ability to audit what happens when something goes wrong.

Lampe says the level of oversight should depend on what an AI system is allowed to do. An AI tool that summarizes information, for example, doesn’t pose the same risk as one that can take action involving a customer’s money or account.

Companies should also continue testing after launch. AI behavior can change over time, meaning a system that performed well when it was first released may not continue performing at the same level later on.

Toyan Espeut, chief customer officer at cloud-based work management and collaboration platform Smartsheet, says companies also need strong rules around customer data.

“It can’t be mixed across tenants or used to train third-party models, and AI can never operate outside a customer’s existing permission structure,” she says. “If a person can’t access something, the AI acting on their behalf shouldn’t be able to either. Every AI action needs to be fully auditable, and customers should always have transparency into when AI is involved, and the ability to review or validate AI-generated outputs.”

Who’s responsible when AI gets it wrong?

Even with extensive controls, companies need to prepare for their AI to make mistakes, but the AI itself can’t be solely to blame.

At Citizen Health, Wang says a problem involving customer trust quickly becomes a company-wide issue. The company has support and on-call procedures, but serious concerns are also brought to leadership. Citizen Health also maintains direct channels for users to report problems so they can be investigated quickly.

Tushman at Hi Marley makes a distinction between the person who initially handles an AI problem and the executive ultimately responsible for it. A customer success or help desk team might receive the first complaint, but he says accountability for customer-facing AI rests with him as CTO. “It isn’t something you outsource to the system or hand to the vendor,” he says.

Corteva Agriscience’s Bhaskar similarly says the company deploying the AI remains responsible for what it does since AI is a tool rather than a decision-maker, and companies can’t shift responsibility to the technology when something goes wrong.

And Lampe adds that every customer-facing AI system should have a named person responsible for its behavior and performance, along with a clear process for escalating problems, and allow customers to challenge incorrect outcomes and reach a human.

Speed: only one measure of success

The push to adopt AI is unlikely to slow down. But tech leaders say companies need to be careful about the standards those investments work by. Simply counting how many customers use an AI tool may not say much about whether it’s making their experience better.

Espeut says companies should look at whether customers are solving problems faster, making better decisions, and getting better business results.

“Companies should also expect fewer support tickets because that tells you customers are actually succeeding with the tool, not just using it,” she says. “Expanded engagement across the organization is also an indicator instead of staying siloed in one buying center. They should pay attention to the quality of customer feedback and feature requests as well. When customers actively help shape your product, it’s a strong signal they trust the technology and see its long-term value.”

Most enterprises govern AI the way they would any other IT rollout: deploy dashboards, institute policies, and conduct periodic reviews. Having the right AI governance tools in place and addressing gaps is important, but IT executives should be more concerned about rethinking workflows and establishing ownership.

“Accountability usually breaks down first,” says Beena Ammanath, executive director of the Global Deloitte AI Institute. “Teams deploy AI faster than their organizations can define who owns risk, approvals, monitoring, and outcomes,” she adds, which results in inconsistent use, fragmented controls, and limitations in scale.

Enterprises need to redesign jobs and workflows, not just the tooling, yet 84% of organizations the Institute surveyed haven’t adjusted jobs for AI, and just 21% have a mature AI governance framework in place despite rapid adoption, according to Deloitte’s 2026 State of AI in the Enterprise report.

“Technology can provide visibility and controls, but it can’t replace leadership,” says Rajesh Arora, chief data and analytics officer at Principal Financial Group. “The organizations that scale AI most effectively are the ones that treat accountability as a business responsibility, not a technology responsibility.”

One year ago Principal was focused on building its ethical and responsible AI framework, its inventory, and governance processes to support responsible adoption. Today the firm has assessed nearly 200 use cases. “Governance is increasingly embedded in those investment and operating decisions, rather than applied as a separate review at the end,” he says.

Costly, compounding mistakes

Some organizations have already run into issues with respect to inadequate governance. A 2024 Stanford RegLab study found that AI-assisted research tools offered by LexisNexis (Lexis+ AI) and Thomson Reuters (Westlaw AI-Assisted Research and Ask Practical Law AI) hallucinated between 17% and 33% of the time when working on legal queries, while the Damien Charlotin AI Hallucination Cases Database has identified some 2,041cases of fabrications so far. In at least one legal case (Couvrette v. Wisnovsky) this resulted in nearly $95,000 in sanctions.

Cases like this one are likely the result of an accountability failure. It may be that no one actually owned the job of verifying the AI’s output before it went to the court. But if a single instance of a static AI output could get past a review today, how will organizations deal with autonomous, multi-step agentic AI systems going forward?

Most use cases at Principal are still human-led, with AI providing decision support only, says Arora. “But we’re preparing for a future where agentic and more autonomous capabilities become increasingly common, and those systems raise new questions around accountability, monitoring, and control,” he says. “The goal isn’t to predict every future use case, but to build adaptable governance processes that can support innovation while maintaining trust and transparency.”

Agentic automation raises the stakes

When things go south with agentic AI today, the consequences usually aren’t devastating because in most cases, the outputs are designed to provide insights, rather than take automated actions. That’s already starting to change, however.

Today, many AI failures are like the one described above where someone failed to catch a single bad output. But agentic AI will gradually take more control of workflow tasks in the enterprise, according to Forrester Research. As those agents act autonomously, crossing multiple process steps along the way, small errors will compound before any periodic review can catch them, so governance must be built into every step in the process.

According to Microsoft Research’s VeriTrail project, catching hallucinations in a single final output isn’t sufficient in multi-step AI workflows. The provenance must be traced back through the intermediate outputs, because an agent chains one call into the next, and an error early in that chain can propagate through several steps before a human sees the finished result. Traditional review, built to check outputs rather than the chains that produced them, isn’t designed to catch that.

So business teams can’t be accountable for agent deployments if they can’t verify performance at runtime. “If the agent is goal-oriented, there could be errors in the workflow process that indicate unwanted behavior or skipped steps that the business wouldn’t be able to detect if they only look at the outcome,” says Lauren Kornutick, senior director analyst, AI governance at Gartner.

Power management company Eaton, for instance, has already deployed multi-step agentic workflows without a human in the loop. “We tend to be risk-conscious, not risk-averse, about autonomous workflows,” says Ross Schalmo, the company’s VP and chief data and AI officer. For example, one autonomous agent responds to supplier payment status inquiries by authenticating the requestor, matching purchase orders and invoices, and returning payment information.

Schalmo’s team engineers such applications to be as deterministic as possible, while the non-deterministic, LLM parts of the program are isolated and continuously evaluated against expected outputs to catch model drift. “When there’s a human in the loop, we can be a lot more permissive,” he says. “Anything autonomous, without a follow-on human step takes longer to develop because we have to think about where to use the generative portion of AI versus where we can still use deterministic tooling.”

Breakthru Beverage Group is still early in its AI journey, but EVP and CIO Glenn Remoreras is well aware of the issues around autonomous agentic AI. “It’s people, process, and culture first — to me that’s the bigger play,” he says. The beverage wholesaler has updated use policies, instituted mandatory AI e-learning programs, and is blocking use of unsanctioned AI tools.

But to successfully manage active, agentic AI, where the AI takes actions autonomously across multi-step workflows, users will need more than basic training in AI, says Craig Le Clair, VP and principal analyst at Forrester Research. “AI will take on more of the work and move the human out of duct tape roles that patch around vague systems, poorly designed interfaces, and poorly integrated data,” he says. “The end game is the agent managing and orchestrating the process — that’s the redesigned operating model.”

Humans, of course, will still need to play a role, although today many employees don’t yet have the required skill sets, and currently that point isn’t getting a lot of attention.

AI supervision skills will be needed for agentic oversight, threshold management, confidence intervals, and root cause analysis, for instance. “Figuring out why an agent hallucinated or got stuck in an execution loop — whether due to poor input data or contradictory rules — and how to refine instructions, that’s the direction,” Le Clair says.

Who’s governing anyway?

AI governance is also complicated by the fact that AI projects exist both inside and outside the IT organization. That leaves governance work distributed across CIOs, data and analytics leads, AI leads, IT security, legal, and other players, with no consistent owner, says Kornutick. “It becomes a multi-role orchestration problem of how to govern this, who owns this, and who’s responsible for which component,” she says.

In response, some organizations have brought in a chief AI officer. Twenty-one percent of organizations now have already done so, says Le Clair, adding that roughly 24% of enterprises will move AI and automation governance outside the CIO’s office over the next year to deal with the risk of fragmentation.

Arora, who reports to CIO Kathy Kay, is the focal point at Principal. “My team establishes the standards, oversight, and governance process, but AI is too embedded in the business to be owned by a single function,” he adds. They collaborate with leaders from the business as well as the legal, compliance, risk, and technology organizations.

The challenge, therefore, lies in deciding who’s responsible for controls, model sprawl, and token economics. “We’re seeing a cooperative, co-model where IT focuses on the hard rails — cybersecurity, performance management, infrastructure reliability, and availability,” Le Clair says, while the CAIO focuses on soft rails, such as context engineering, the effects on employees as agents move to the center of the process and start taking over the middleware function humans used to perform, and connecting disparate systems and poorly designed interfaces.

At Eaton, governance is built on a three-legged stool between Schalmo’s organization, cybersecurity, and legal, while enterprise architecture weighs in on tech foundation decisions. The company also assigns ownership of an AI project only after a successful pilot, while IT performs a custodian or stewardship role. “Once that agent is live and operational, it shows up in an org chart, reporting to someone in the organization,” he says.

At Breakthru Beverage, Remoreras formed and leads an AI governance team that reports to the board and co-leads with the VP of data and AI a governance council that reviews and approves AI projects from SVP- and VP-level product owners. “I’m seen as the owner of our AI portfolio, but it’s co-leadership with my peers on the executive team,” he says.

Closing the tools gap

With regard to AI management platforms, says Le Clair, none of the platforms building AI agents have an adequate control plane today to give the visibility needed to move to truly agentic solutions. The most advanced planes are walled gardens. For example, Microsoft’s agent registry will show every agent built across its own platforms, says Le Clair, but not agents built on Salesforce or AWS.

“You’ve got maybe 50 vendors trying to be pure-play control-plane providers, plus the hyperscalers, portfolio companies, and startups building agentic frameworks,” he adds. “There’s no consensus yet on what’s even most important to control, and there’s confusion about requirements.”

Most AI governance tools available today are limited, Schalmo says. “We’ve looked at four or five different external vendors for AI governance and AI control-tower-type solutions, and a lot of them fall down on discovery of AI built by someone else’s solutions,” he says. “That’s a huge gap for me. Either I have to buy governance solutions from almost every piece of software we interact with, which gets really expensive, or I have major gaps in my portfolio I just don’t have visibility into.”

Currently, he’s looking at using a cyber vendor’s AI component for discoverability, going with at least one major firm that provides control-tower capability, and using AI to build another governance layer on top of that.

Organizations clearly need agent registries and agent lifecycle management but it’s less clear what they require beyond that. “How deep does explainability need to go when decisions are made at runtime that weren’t articulated at design time?” Le Clair asks. “There’s real vagueness in the requirements. Vendors have very well-tuned pitches that promise to do anything you want, but buyers often don’t know what they need in the first place, and there are real tool gaps. This is the major problem the industry is actively trying to solve. It’ll just take time.”

But those aren’t Arora’s biggest concerns. “Most organizations can acquire tools,” he says. “Far fewer have clear ownership, decision rights, and accountability embedded across the business.”

Speed versus governance

There’s a perception that governance slows down AI projects because it can’t keep up with the rapid pace of adoption and advancement. A 2026 GitLab AI accountability survey of developers and technology buyers found that 92% of organizations reported some form of governance challenge with respect to governing AI-generated code, and 34% of organizations that experienced an AI-related incident couldn’t determine after the fact whether AI-generated code was the cause.

Similarly, Faros AI’s telemetry analysis of 22,000 developers in its 2026 Engineering Report discovered that developers with high AI adoption rates completed 34% more tasks and 66% more epics, but the median pull-request review time rose by 441%, a nearly five-fold increase.

Is it the case that governance and speed are inherently at odds, and that human review simply can’t keep up? That’s a false choice, says Ammanath. Rather, it shows what happens when both aren’t designed together. “When done well, governance accelerates scaling by giving teams clear guardrails from the start,” she says. “What slows organizations down is retrofitting governance after broad deployment.”

Arora adds that controls don’t slow down innovation but protect the conditions that allow innovation to scale responsibly. “Our approach has always been to move with urgency, but not at the expense of trust,” he says.

How to proceed

For Principal, investing in workforce readiness was one of its biggest investments, helping employees understand both the opportunities and responsibilities that come with these tools, Arora says.

But organizations shouldn’t stop there. Redesigning workflows and decision-making structures is essential before scaling, says Ammanath. “Many organizations have policies and technology in place but lack clear decision rights or governance embedded into daily work,” she says.

Operating model changes take more time than training, and require executive sponsorship and cross-functional coordination. “Without workflow redesign, a workforce may understand AI but lack clarity on when to use it, how to govern it, and how daily responsibilities change,” she adds. “The organizations seeing the strongest results pair broad AI education with role-specific training, workflow redesign, and clear accountability for outcomes.”

Make sure you’ve thoroughly considered the organizational change management aspect of agentic AI projects, says Schalmo at Eaton. His team developed an automated workflow for processing complex bid requests to save the sales team time, but piloted it in a country with lower labor costs than the US. That challenged the economics of the project, and those individuals only dealt with higher-volume, lower-complexity cases.

The lesson: “Lead from a people-first standpoint, and understand the workflow and what’s impacted — whether it’s relieving drudgery and freeing someone for higher-order thinking or customer time, or impacting an actual support function’s role,” he says. “And make sure there’s a viable path to business value.”

An adaptive governance process is key to keeping up with the technology, Kornutick at Gartner adds. She recommends categorizing AI projects following a stoplight approach. Green light projects, such as for personal assistants, present acceptable risks. “We know there’s risk associated with it, but we’re willing to take it,” she says. Yellow-light projects need guardrails in place around such things as PII, or for those triggering a privacy-by-design process, or a security review of a new application. Red light projects need an architectural review before proceeding.

“Think about those use cases as business-transformation processes where you’re fully redesigning workflows, or considering replacing tasks individuals do and shifting them to supervising the AI doing those tasks instead,” she says.

Schalmo’s team does risk tiering based on the EU AI Act’s four-tier risk framework to determine the appropriate level of governance scrutiny for each project. Low-risk requests are automatically provisioned after the requestor fills out a questionnaire, while high-risk cases get a full architectural review.

“Treat governance as the foundation for scaling, not a constraint on speed,” says Ammanath. “Before expanding agentic AI, organizations should define what decisions agents can make, where human oversight is required, who’s accountable, and how performance and risk will be monitored.”

The hardest part, says Arora, is helping an entire organization evolve at the pace of technology. “Successful AI adoption is ultimately a business transformation challenge,” he says. “It requires leadership alignment, clear accountability, workforce readiness, strong governance, and ongoing change management. Long-term trust will belong to organizations that can translate technology into sustainable business value.”

AI adoption in the enterprise is way up — and so too are IT leaders, into the night, dealing with the risks and governance issues of AI use.

To be sure, increasing attention to AI governance is a welcome addition to enterprise AI strategies, shifting an anything-goes approach toward risk-aware plans of action directed toward value. But it’s also taking an unspoken toll on those responsible — with the challenges of agentic AI mounting fast.

Not only are four in five senior business decision-makers, including CIOs, CISOs, and CDOs, using more of their day to manage AI risk, but their working hours are up 26% on average due to the issue, according to a new survey released by AI governance platform vendor OneTrust.

Contrast that to recent findings from Boston Consulting Group, which found that 42% of frontline employees who regularly use AI save nearly a full day of work each week, the majority of whom are given no guidance on what to do with that time saved, and you get an organizational picture of rampant activity, questionable direction, and lots of high-salaried time spent on cleanup and oversight.

Much of leaders’ extra time spent on managing AI risk is due to an increased awareness of AI risk and governance, IT leaders and industry observers say, but triage and other factors play a significant role as well.

For example, a third of respondents to OneTrust’s survey say they’ve seen employees use unapproved AI because approved tools or processes weren’t available quickly enough. Eighty-six percent have reported AI-related incidents, and over a quarter have experienced two or more incidents of AI systems taking unapproved actions.

However, as Blake Brannon, chief innovation officer at OneTrust, sees it, a huge part of the equation is AI adoption, with use skyrocketing.

“You’ve got the sheer volume and adoption of AI in organizations,” he says. “Whatever AI they were using at the beginning of the year, it’s probably more than 2x what they’re now looking at using inside the company, and that’s a compounding thing.”

Governing citizen coders and unsanctioned use

One key factor is increasing permission for non-IT employees to use AI to build their own apps, Brannon says. That move toward citizen development sparks a greater need for governance, he adds.

“We’re seeing exponential growth, not necessarily in different AI providers or tools, although that is growing, but the explosion of citizen builders and the new creativity that has been unlocked,” he says.

AI is creating new challenges for IT leaders as fast as they can understand them, he says, and the technology complicates risk management because of the speed at which it can make decisions and create problems.

“Security, compliance, risk management historically have just had to govern humans,” he says. “Humans were deterministic, and they moved at human speed.”

Viren Meghani, technical architect at Tata Consultancy Services, also sees IT leaders spending more time on AI governance and risk management than a year ago, but believes that’s largely related to shadow AI.

“Interestingly, this increase isn’t coming from overseeing the approved tools,” he says. “Instead, it’s all about chasing down the unapproved ones.”

Meghani believes employees’ urgency for AI solutions is a key factor. “People don’t wait around; they find a tool that works for them,” he says of unsanctioned AI use. “Then, governance has to scramble to put controls in place around something that was never properly evaluated.”

Spending more time on governance also doesn’t mean IT leaders are doing it correctly, he adds.

“Time spent reacting to incidents doesn’t equate to time spent on building effective controls,” he says. “If you’re busy investigating what an unapproved tool did, you’re not really governing; you’re just putting out fires.”

The organizations taking AI governance seriously invest time in architecture before deployment, using methods such as data lineage, model versioning, escalation paths, and audit trails, Meghani explains. “That kind of work might not grab headlines, but it’s what really helps to lower the number of incidents,” he adds.

Awareness isn’t readiness

Conal Gallagher, CIO and CISO at IT management software vendor Flexera, also believes taking governance seriously doesn’t mean organizations have achieved AI readiness.

“The risk surface of AI has expanded so quickly that the time designated to managing that risk continues to increase,” he notes. “Just one year ago, much of the focus was on enabling employees to experiment with AI, but now we’re seeing cases where autonomous agents create other agents and identify vulnerabilities at incredible speed.”

Governance is constantly playing catch-up to the changes happening with AI, Gallagher adds.

“Our IT team’s approach is to be judicious about which capabilities we put into employees’ hands and to slow deployment when we don’t understand the risk well enough,” he says. “We’re also applying established security principles to make sure agents don’t have greater access to capabilities than they actually need. The technology may be new, but some of the most important controls are still familiar ones.”

Anant Adya, EVP and head of Americas delivery at Infosys, says he’s spending more time managing AI risks now than a year ago, a natural shift, he adds, given the company’s move from experimentation to enterprise-scale adoption.

“While it does require additional attention, it is becoming embedded into our operating model rather than simply adding hours to the day,” he says of Infosys’ added emphasis on governance, security, privacy, compliance, and responsible AI practices.

That effort to operationalize governance and control is vital as the technology moves into core business workflows, Adya says, especially as agentic AI — what’s next — will only add another level of urgency.

“As these systems gain greater autonomy, they can access sensitive data, interact with other systems, and take actions on behalf of an organization,” he says. “Enterprises need strong identity controls, continuous monitoring, and appropriate human oversight.”

The struggle for good governance remains an issue for many organizations, but CIOs facing an overwork problem playing catch-up on AI governance are advised to get a handle on enforcing what matters before autonomous workflows compound the challenge.

During a recent architecture review with a client, I asked their leadership to trace a single automated transaction backward through their production systems. Three days earlier, an embedded customer support agent had issued an unapproved account credit to a corporate client. Every system monitoring dashboard was glowing green. Internal network logs showed a clean, successful transaction. Cloud performance monitors showed standard processing times.

When we queried the core financial system, the control environment could not answer the question we were trying to resolve. There was no log explaining what customer context triggered the refund, what calculations took place, or which corporate business policy authorized the spend. The software had executed cleanly from a technical standpoint, but from a business governance standpoint, it was an unauthorized transaction.

It is easy to assume that an agent inherits some of the trust of the application it lives inside. It doesn’t. A secure Salesforce, SAP, or Workday environment still does not answer whether an agent was authorized to make a particular business decision. If it makes the wrong move, the vendor does not own the resulting business problem. The enterprise does.

According to research from Gartner, forty percent of enterprise applications will be integrated with task-specific AI agents by the end of 2026, up from less than five percent in 2025.

That forecast will likely be read as proof that enterprise AI adoption is accelerating. But it measures something narrower: how quickly vendors are putting agents into products companies already use. Software can be deployed long before the organization has decided what that software is actually allowed to decide. That forecast tells us how quickly agents are arriving. It does not tell us whether the organizations deploying them have decided where their authority begins and ends.

The software procurement trap

The subscription price on the vendor’s order form is only the first line item.

Enterprise procurement teams are still buying autonomous software agents as if they were traditional application features, evaluating additions using familiar checklists: seat pricing, uptime guarantees, data encryption and user access roles.

The moment software can initiate payments, adjust contract pricing, alter supplier terms, or reroute customer orders without a person reviewing the choice first, an organization has delegated part of its business authority to software. At that point, seat price is only part of the evaluation. The real issue is what authority the company has handed over.

If an agent can execute consequential decisions, the enterprise also pays for the operational guardrails around it: ongoing monitoring, policy enforcement, historical audit logging, incident response and the manual finance or legal cleanup required when something goes wrong. An agent that looks inexpensive during contract negotiations can become remarkably expensive to support in daily operations.

Monitoring is not authorization

In architecture reviews, I keep seeing the same four questions get mixed together:

• Monitoring: Is the system working?
• Auditability: Can we reconstruct what it did and why?
• Authorization: Was it allowed to do it?
• Accountability: Who owns the consequence?

A software vendor’s cloud security certification proves that its underlying infrastructure is protected against outside intruders. It does not prove that an action taken by an automated tool complied with your internal company rules. If an embedded sales agent commits your organization to an unapproved contract discount, the vendor’s security report remains valid, but your revenue margin takes the hit.

Technical performance monitors verify system mechanics, not business permissions. An operations dashboard can confirm that a request finished in 240 milliseconds, but it cannot tell an internal auditor whether the system should have approved the credit in the first place. Traditional enterprise software follows rigid rules where decision paths are mapped out in advance. Autonomous agents interpret unstructured information and choose among possible actions on the fly.

NIST’s 2026 research identifies fragmented logging across distributed infrastructure as a monitoring problem and points out that the relationship between monitoring and auditing is still unresolved. That matters because an enterprise can have healthy systems and extensive logs while still being unable to prove that a business decision was authorized.

Mandating that human employees approve every automated action does not solve this problem at enterprise scale. Human-in-the-loop workflows look safe in leadership meetings, but they break down under transaction volume. Give an employee five flagged exceptions a week, and they investigate each one carefully. Give that same employee two hundred automated approval requests a day, and clearing the backlog becomes the primary job. The review turns into a routine rubber stamp. Human approval only works as a control when the person approving the action has the time and context to evaluate it.

Conversely, forcing senior managers to manually double-check every piece of background information behind every automated suggestion destroys projected productivity gains. You end up paying the full subscription cost for the automated system while retaining the full payroll cost of manual review.

When nobody owns the decision

The AI failures that make headlines are usually the obvious ones. Enterprise systems have another class of failure: the transaction that succeeds.

Traditional IT monitoring is designed to catch systems that break. A server stops responding, a database times out, or an application crashes, triggering an immediate alert. Policy failures do not trigger technical alarms. The purchase order processes, the supplier gets paid, the customer receives a confirmation email and the transaction closes. From a technical standpoint, nothing failed. From an internal governance standpoint, it was a complete failure of controls.

Consider how this plays out in daily operations. In procurement, an automated purchasing agent routinely routes material orders to a preferred supplier that delivers quickly, quietly bypassing a corporate policy that requires gathering three competitive bids for purchases over $50,000. The same thing can happen elsewhere across the business. A sales agent can offer custom payment terms that violate internal accounting rules for revenue recognition. A customer-support agent can resolve an urgent ticket by pulling private client records across department lines without proper authorization.

In every case, the software ran as intended, the task finished and the operations dashboard displayed green checkmarks. Yet an automated process changed company records or decisions outside authorized policy.

The second Gartner forecast changes the conversation. Gartner predicts that by 2027, forty percent of enterprises will demote or decommission autonomous AI agents due to governance gaps discovered only after production incidents occur. The research specifically points to the failure to distinguish an agent’s ability to act from the scope of access it is granted.

An agent may possess the technical capability to update a database record without the enterprise ever making an explicit decision that it should be allowed to do so. That problem compounds when workflows cross multiple vendor applications. If one software platform defines permissions one way, and your core enterprise database defines them another, who owns the business rules when an automated workflow spans both?

The vendor can provide the software, but the enterprise still owns the business rules.

In my work evaluating production architectures, I repeatedly see governance fall into an ownership vacuum. The business application team assumes cybersecurity is managing agent permissions. Cybersecurity assumes the business process owner defined the operational rules. Finance assumes the software vendor engineered the platform to prevent policy violations.

By the time leadership asks who authorized the behavior, the system may have been running for months and making decisions nobody explicitly approved. Once money has moved or an unvetted contract term has been delivered to a client, an audit log can only document what happened. It cannot make the decision authorized.

Enterprises have delegated authority to software for decades through scheduled batch jobs, service accounts and automated scripts. What changes with autonomous agents is that the software has far more latitude to decide how it accomplishes an objective. Asking “Who has access to the application?” is no longer enough. You must also ask: “What decisions is the software allowed to make with that access?”

What changes for the CIO

The answer is not to put an approval committee in front of every automated action. That would simply replace one problem with another. Applying uniform, manual controls to every automated feature is just as flawed as deploying agents with no guardrails at all. The more consequential the action, the stronger the control should be.

A meeting-summary agent and an agent that can issue a refund should not operate under the same control boundary. A software provider can update an agent’s underlying behavior during a routine update without changing your enterprise policy. For that reason, business authority rules cannot live solely inside the vendor’s application.

For high-consequence actions, the enterprise needs a separate check against its own policies and financial limits before the record changes. The evidence detailing why that action was permitted must be captured independently, ensuring that when internal auditors or regulators examine the transaction a year later, the enterprise can verify both the context and the business authorization behind it.

Before approving the rollout of embedded autonomous agents, every leader should put six practical questions to their procurement and architecture teams:

1. Which vendor-supplied agents can currently modify live business records, alter contracts, or execute financial transactions?
2. What access does each agent receive from the user, and what additional access does it have on its own?
3. Which named business leader owns and audits the operational rules governing each agent?
4. How do our controls distinguish low-risk informational tasks from high-consequence write operations and financial commitments?
5. What happens to our governance boundaries when a vendor updates an agent’s underlying decision logic in a routine software patch?
6. If an auditor asks six months from now why an automated decision was made, can we prove both the business context that triggered it and who authorized it?

Software vendors will continue embedding autonomous agents into the enterprise applications companies already use. C-suite cannot let that release cycle become the company’s authority model. The organization still has to decide what those systems are allowed to do and be able to prove that decision later.

While its flagship Dreamforce event was in full swing on Wednesday, Salesforce was triaging a roughly seven and a half hour-long service outage that disrupted user access and caused “severe delays” and intermittent errors. Some customers were also unable to submit new support cases.

The service outage hit at 3:50 a.m. EDT, impacting “multiple instances across all regions,” Salesforce reported. It was marked resolved right around 3 p.m. EDT, after several hours of monitoring to determine that fixes had been successful.

Salesforce initially pegged the issue to an “external dependency failure” impacting the legacy login server. A core system component experienced increased load, limiting its capacity to process requests. The company confirmed that there were no issues with third-party infrastructure.

Beyond the obvious embarrassment from the outage occurring during Dreamforce, an analyst said the incident highlights a cloud resilience problem, rather than purely a legacy one.

“Cloud does not eliminate architectural dependencies,” said Abbas Jaffery, a principal advisory director at Info-Tech Research Group. “It can sometimes make them less visible. And when the platform is your system of record, those hidden dependencies become an enterprise risk rather than simply a technology risk.”

Rolling restarts, persistent issues throughout the day

Salesforce began experiencing service issues at 3:50 a.m. EDT on September 16, and initial investigation determined that requests were stalling while waiting on responses from an internal login service that was using up available server resources.

Initially, Salesforce blocked application programming interface (API) endpoints and attempted rolling restarts to revive the service, then pushed out fixes region-by-region. At 7:20 a.m., some customers were seeing service return to normal, and Salesforce was working on a “code-level permanent fix.”

However, the rollout did not complete for a number of instances, and some automated fixes didn’t fully resolve the issue. For example, customers reported that scheduled jobs were not running as expected, even after service was restored. Salesforce manually restarted those ‌instances.

Salesforce later reported that the impact radius was “narrower than initially understood” and saw signs of recovery by around 11 a.m. EDT, with most customers coming back online.

A sub-set of Hyperforce instances were the last to be restored. Mitigations were in place across all instances by 11:39 a.m. EDT, and Salesforce continued to monitor the issue until marking the incident resolved at 2:59 p.m. EDT.

“We apologize for how this incident affected you and your business,” Salesforce posted on its incident blog. “We will undertake a full investigation of the incident, establishing the technical trigger, the underlying cause, and preventive action to avoid a repeat in the future.”

Creating a ‘temporal’ data problem

For customers for whom Salesforce is a system of record, several hours of authentication and service disruption can create a “temporal data problem,” Info-Tech’s Jaffery explained. “Events that should have happened at different points in time may occur later, fail altogether, or arrive out of sequence,” he said.

For instance, a customer interaction may occur through another channel while Salesforce is unavailable, but an integration, workflow, or scheduled process that normally records or propagates that event is unable to run.

This has several potential consequences, Jaffery said. Transactions and customer service processes are delayed; APIs and middleware may accumulate retries, timeouts, and queues. Records can become temporarily inconsistent, and scheduled jobs and workflows may be missed. Employees could lose visibility into customer history or case status, even when the underlying data has not been lost, creating a “data divergence.”

The first mistake would be to assume that just because users can log in, the incident is over, he noted. “Enterprises should move immediately into a reconciliation and integrity phase,” Jaffery advised. This means not only verifying interactive access, but APIs, integrations, scheduled jobs, queues, workflows, automation, authentication flows, and downstream systems.

Enterprises should ask what transactions failed, partially completed, or were duplicated during the outage? Which scheduled or asynchronous processes did not execute? Did integrations retry successfully, or did they create a backlog or retry storm? Are downstream systems now consistent with Salesforce?

Security teams should also validate authentication and session behavior, privileged access, integration credentials, and any emergency changes made during recovery, Jaffery explained. “The most important question is not simply ‘Is Salesforce back?’, but ‘What did the business expect to happen during the outage, and can we prove that it actually happened?,’” he said.

What to look for in post-incident reports

A credible post-incident review from Salesforce should establish a causal chain: The trigger, dependency failure, technical propagation, customer impact, detection, mitigation, recovery, and permanent corrective action, Jaffery said.

The company should be able to answer these questions, he said:

• What was the actual initiating failure and why did the failure propagate into the login path?
• Why could the affected dependency consume sufficient capacity to affect core services?
• Why didn’t isolation or failover prevent the impact?
• Why did initial remediation attempts fail and why did the subsequent rollout require additional intervention?
• What safeguards are being added to prevent recurrence?
• How will Salesforce demonstrate that the corrective action actually works under failure conditions?

Service restoration simply tells customers: “We got it working again,” he noted. But root cause analysis tells customers: “We understand why it failed, why our controls didn’t prevent it, and what has changed so that the same failure mode is less likely to recur.”

It’s not just about ‘legacy’ pieces in the stack

One architectural lesson is that a legacy component does not have to be large to be critical, Jaffery pointed out. An older authentication service can remain part of a modern stack, and therefore become a dependency for newer services.

“The component’s age matters less than its position in the dependency graph, its blast radius, and the quality of its isolation and failure handling,” he said, pointing to this incident’s progression: Requests stalled waiting on an internal login service due to increased resource consumption led to investigation into an external dependency failure, which in turn revealed impact on a legacy login server. Finally, Salesforce said, “core system components experienced increased load, which limited its capacity to process request”.

That is a classic resilience question, Jaffery pointed out: Can a failure in one dependency remain in that one dependency, or does it become a platform-wide failure?

Modernization should not be identified simply by how much old technology has been replaced, he noted, it should also measure dependency concentration, isolation, “graceful degradation,” recovery paths, and failure blast radius.

“For enterprise architects, that is the real takeaway,” he said.

Maybe driven by agentic AI, exacerbated by layoffs

At this point, there are no obvious signs that this was a security incident, noted David Shipley, CEO of Beauceron Security. “Right now, this bears all the hallmarks of an update gone horribly wrong.”

He pointed to an incident in December 2025 when Amazon’s internal AI coding agent, Kiro, caused a 13-hour AWS outage in a mainland China region, noting, “I’m not going to be shocked if we don’t see some kind of agent role in this kind of scale disaster.”

Significant Salesforce layoffs over the last few years could also have had a negative impact on the outage and recovery, he added. “Having it happen during Dreamforce had to be all kinds of hell, though, for their sales and customer support teams,” he said. “Pour one out for them as they work on rebuilding relationships, face-to-face.”

Organizations are racing to deploy AI, but warning signs are accumulating. Earlier this year, an internal AI agent gave an engineer instructions that exposed sensitive user and company data for two hours. Around the same time, a large online retailer issued a 90-day safety reset after its AI assistant contributed to an incident that involved nearly 120,000 lost orders. And in the spring, an AI agent deleted a company’s production database and its volume-level backups in nine seconds.

Over recent years, recurring events like these, among others, expose a widening gap between what AI can do and what organizations can safely control.

“A year ago, most conversations were about accelerating AI adoption as fast as possible,” says Sandeep Johri, CEO at application security platform Checkmarx. “Today, boards ask tougher questions. Speed and governance have to move together now.”

As Johri points out, the new bottleneck is the organization’s ability to govern AI. According to IBM’s 2026 Tech Leader Study, 77% of organizations admit their governance is failing to keep pace with AI. And, among IT executives, 70% say business teams are deploying tech faster than it can be tracked.

The use of agentic AI only widens the gap. About 80% of the organizations surveyed say they lack mature capabilities for it, according to Deloitte. That includes clear boundaries for agents, real-time monitoring systems, and audit trails that can capture the entire chain of actions.

CIOs need to operate in this paradigm to address two competing demands: accelerate AI adoption to boost productivity and outsmart competitors, and assure boards that all sensitive data is protected and AI only does what it’s supposed to do.

“I don’t think you can separate the two,” says Sahil Sanghvi, VP of AI engineering in the chief technology office at Booz Allen Hamilton.

Innovating while managing risks

At first glance, AI-generated code can look good and even pass initial testing. A thorough review, however, can shed light on multiple issues. This is something Ha Hoang, CIO at data protection platform Commvault, witnessed firsthand.

In one case, her team found the AI had taken a shortcut. It bypassed the company’s authentication process in favor of a simplified implementation, which lacked established access controls. “Without those checkpoints, it could’ve made its way much further,” she says.

When companies discover major issues, they should immediately pause deployment. But many problems aren’t obvious. “AI-driven risks often remain hidden, and moving too quickly only makes those silent failures harder to detect,” says Omer Cohen, CISO at customer identity and authentication service Descope.

But to strictly move slowly everywhere isn’t an option either. The idea is to identify where speed creates value, and where the potential consequences call for caution, and then build necessary guardrails case by case.

For Bob Leek, CIO at Clark County, Nevada, that means making governance and compliance part of the design, not a final check before deployment. “We’ll go slow to go far instead of going fast and creating risks,” he says.

The biggest challenge is organizational, not technical

In many cases, AI deployment is less a technology problem than a people problem. When deciding what to automate inside an organization and how to do it, the real challenge is understanding how work actually gets done. And usually there are many invisible, undocumented processes that influence it.

Employees in HR, finance, procurement, legal, or operations rely on exceptions every day. They have workarounds and make judgment calls to keep the organization running. These tweaks are simply part of the job, so they rarely think about them or include them in official process documentation.

These elusive workflows can’t be mapped simply by considering how things are supposed to work. Leaders must closely observe how employees actually do their jobs.

“Frontline teams understand the exceptions, escalation paths, and context that rarely appear in a process map,” says Leek. “We bring those teams into the design process, mapping the handoffs and non-standard cases.”

Cohen agrees. “Invisible threads are often fragments of context residing in an individual’s mind rather than a database,” he says. For instance, an analyst may know that a client’s login spike is harmless because it’s scheduled during weekly testing. “Unless this tribal knowledge is codified as a formal governance artifact via runbooks, threat models, or decision logs, no AI will naturally possess it,” he adds.

But simply asking employees how they work isn’t enough, adds Amitkumar Rathi, chief product and technology officer at hybrid infrastructure observability platform Virtana. The best approach is to run shadow sessions, in which someone in tech actually witnesses how the work is done. “We sit next to them during live incidents and ask, for instance, why did you look at that dashboard and not this one; why escalate now and not 10 minutes ago; what told you this was the same issue as last month’s incident and not a new one?” he says.

Of course, mapping informal processes takes time and discipline, and there shouldn’t be any tempting shortcuts. “The organizations that get this right treat AI as a collaborator in their existing workflows, not a replacement,” says Vijay Jegan, chief AI transformation officer at enterprise customer retention platform Gainsight. “Success requires a hybrid of deep business acumen within a department and the technical maturity to understand the inherent risks of modern AI tools.”

But not all tribal knowledge can or should be documented. “The goal should be to architect AI to augment this human foundation, rather than attempt to replace it entirely,” adds Cohen.

Where should humans stay in the loop

Giving AI a larger role makes human judgment more important, not less. “Humans should stay in the loop in every decision, but not every part of the process,” says Leek. “The urgency to innovate doesn’t change that fundamental responsibility.”

CIOs can decide where people should remain involved by weighing the value of human judgment and the risk of leaving the task entirely to AI. Tasks that score highly on both should remain firmly in human hands. “The higher the risk, the more human oversight is required,” Jegan says.

Sanghvi also factors in human consequences of potential AI mistakes. “When you deal with a decision that could materially affect a person, a mission, or an organization, that’s where you want clear human authority to intervene or override the system,” he says. “As AI becomes more agentic and starts taking actions rather than just making recommendations, being clear about those boundaries becomes even more important.”

Meanwhile, Cohen draws the line at AI-powered decisions that can’t easily be undone. “Human intervention remains non-negotiable at any juncture where a decision becomes irreversible or traverses a critical trust boundary,” he says.

At the other end of the spectrum, routine, low-risk work can be left to the machine. “Organizations may trust agents to autonomously handle narrow, repeatable tasks,” says Hoang, adding, though, that even advanced agents can misinterpret context or take unintended actions at scale.

“The future isn’t blind trust but measurable trust built on transparency and control,” she says.

Governance doesn’t end at launch

Before an AI initiative becomes a major commitment, Leek recommends CIOs ask if the project supports the organization’s strategic priorities, if IT can support it, and does the business department have the capability and appetite to change?

“This framework helps prevent initiatives from becoming solutions in search of a problem,” he says. It also helps CIOs start with lower-risk projects, test what works, and strengthen governance before applying AI in more sensitive areas of the organization.

Clark County took that approach with its first AI deployment for special-event permitting. Its AI tool guides promoter through forms, identifies the permits needed, and connects them with a county analyst. But starting with a lower-risk project doesn’t mean the governance work ends at launch. Governance should be a continuous conversation rather than a checkpoint, says Sanghvi, since data changes and models evolve.

Hoang agrees. “If your governance system relies on quarterly reviews, you’re already behind,” she says.

Over the course of his career, Jim Reavis has seen cloud and cloud security evolve, and it’s come a long way since being a niche technology in the early 2000s. Now it’s dominant in terms of being the IT foundation, he says, but while the tech is strong, the operating models is where things get messy. Cloud, security, and third-party risk teams look at different parts of the problem, of course, but challenges remain.

“Operational technology worries me a great deal,” he says. “A lot of those systems are isolated and not kept up to date. If we don’t modernize them, we’re going to have huge problems. In a lot of cases, things fall between the cracks and that’s where hackers like to exist.”

So much of what’s around the models is where cybersecurity has responsibility, rather than the provider covering everything. “Data, identity, and applications are shared responsibility areas, and in many cases, the tenant carries most of the control burden,” Reavis says. “If you use a hyperscaler, you may still have about 80% of the responsibility for the controls around what you build.”

And when it comes to AI, the model isn’t the whole problem. What matters is the context around it, the goals it’s given, and the oversight put in place, he says. “We need to think carefully about the harnesses we put around AI and the systems we use,” he adds.

The responsibility model, therefore, is a recurring issue in cloud security breaches tied to misconfiguration and accountability gaps, and some enterprises still aren’t clear about where responsibility begins and ends. “We spent a lot of time on a shared security responsibility model, but when this first started to gain popularity, there were a lot of organizations or SaaS providers you could work with who’d say it’s in the cloud, it’s at Amazon,” he says. “Look at their certifications and SOC2 and how they comply because they’re covering everything.” But when you look at the actual applications, data, and identity, he adds, there’s so much that’s shared responsibility, and the customer’s responsibility.

So how do we make sure information is encrypted properly so it doesn’t become a tenant issue? “There’s still a bit to do, and we think about this not only from whether it’s SaaS, infrastructure, or a particular provider, but at what level is it at the physical, network, or audit level,” he says. “And even from a role-based perspective, what’s the role of internal risk and role of providers?”

Reavis gives further detail about how AI adoption exposes weaknesses in identity, trust, and risk management, and the long-term implications of increasingly interconnected cloud ecosystems. Watch the full video below for more insights, and be sure to subscribe to the monthly Center Stage newsletter by clicking here.

On cloud risk management: When we had the Chat GPT moment, we knew it because AI had been around for a while, but that was a cloud delivered version of AI to the masses, so we saw this going to evolve and you could see it combining in many important areas.

But what we’ve learned is, because this is an interesting predictive rather than deterministic technology, we’re living in a world of two exponentials, and you’re seeing model capabilities growing so quickly. There’s this feeling from a security perspective that we have to look to the model itself and fix every hallucination and everything else when that’s built into how it works. It’s actually working as intended. So that’s a new lesson. Models are going to get more powerful, but it’s so much of what’s around the models where cybersecurity has responsibility, and we don’t rely on frontier model companies or using open-weight models. Rather, what’s the context, oversight, and information we’re providing them, what do we do in terms of goals we give them, and what are the harnesses we put around AI and the models we deal with?

These are going to be the big areas to think about, but we have to understand the parts we can control. We’ve got to think carefully about the harnesses, transparency, and using supply chain shared responsibility. SaaS and cloud providers are all AI enabled now. You’re not using any software of any significance that isn’t using AI to some degree.

On AI identity, trust, and control: One of the areas that we’ve championed is zero trust as a philosophy. It was initially more of a networking type of approach at the network layer, or an idea that you use identity to understand network access. But it’s evolved more to an idea that anything can be breached, so you assume that. Then you think about how to make systems resilient, and build up confidence and protection.

So zero trust tells us that with human identity, we can ask what our digital identity is, and now we’re in a very interesting area for identity management and associating that with agents and AI systems. People might have just one view of it, but agents are as diverse as humans. So we think about different identities and least privilege, and how to prevent them from escalating privileges. We need to introduce new concepts like least autonomy, and think about an agent that has certain tasks and use identity to make sure the actions it takes are within a defined scope. Because while we’ll see a lot of security incidents with AI, proportionately we’ll see more misconfiguration and bad things that happen because of broken processes. And the AI system just deletes things because it thought that’s what it’s supposed to do.

So it’s important to make strides in how we think about identity and agents, and the idea of digital workers. How do we manage and treat those? If we think about them too much in either one of those realms, we’re going to fail. So we have to understand what’s the right blend. It’s a new area and very exciting.

On risk and legacy systems: When I think about operational technology, sometimes systems are isolated and not kept up to date. That concerns me a great deal. We’re going to have huge problems there. We have concerns about existential risks, where people don’t want to use the latest technologies and be aggressive adopters of AI. I think that’s going to create real scale issues with organizations.

So we have to understand where we are, where we’re going, and have a vision that serves something between human and technology, maybe a hybrid, but we’ve got to make our peace with it and understand the appropriate harnesses and direction where humans should always be in the loop with control. But it’s appearing in some new areas of cybersecurity where we haven’t traditionally thought about. Software development looks very different now than it did 12 months ago, and 12 months from now, cybersecurity is going to be really different, too.

On cloud security and implementation: Cloud security is cybersecurity for all intents and purposes. We have so much tooling and technology that’s really good, but there’s a lot of inconsistencies with the operating models organizations have. Even way back with CSA and NIST defining this, it was clear that SaaS was a layer on top of infrastructure as a service. But we diverged, and you see in a lot of enterprises there’s diffused ownership where you have cloud and security teams, and then you have third-party risk that deals with the SaaS team. Then there are inconsistencies in how risks are managed, so internal development and expectations from our partners can really diverge. They have a lot of regulations to deal with, so it creates vetting and investment challenges while striving for consistent models.

Some security teams might still use older checklists to talk to their cloud teams, but scaling with new tech becomes an issue if you’re not thinking about operations. It ends up being a human and a structure problem that makes it harder to take advantage of all the great technology that’s out there.

Concerns about the risks of AI systems are certain to be met with four words: human in the loop. The discussion may broaden, but the assurance is inevitable. It’s an AI governance phrase that’s become so rote you hear it in every direction and likely have said it yourself.

But IT leaders should be wary of vendor or team claims that they’ve built human-in-the-loop systems into AI tools because some of these supposed guardrails are no more than rubber stamps.

Some so-called human-in-the-loop systems don’t give employees overseeing the AI tools either the control or the time necessary to fix any problems, some IT experts point out.

For human-in-the-loop systems to actually work, employees overseeing AI tools need to have the domain knowledge and context to take the action the AI tool is addressing when the AI isn’t involved, and they need to have the authority to override the AI decision, says Doug Shepherd, head of offensive security at internet services provider Cloudflare.

Promises of human-in-the-loop systems give IT leaders comfort, but the underlying process often doesn’t work as advertised, he adds.

“If your human in the loop can flag something but can’t actually stop it, that’s not human in the loop, that’s a human adjacent to the loop,” Shepherd says. “That’s performative governance.”

Shepherd, speaking at the recent CIO 100 Awards and Conference in Frisco, Texas, encouraged attendees to embrace AI and focus on projects that drive adoption and impact. Organizations that fail to push AI initiatives will be left behind, he suggested, but he also warned that blind adoption, without focusing on meaningful outcomes and guardrails, can lead to huge setbacks.

Many organizations reach for human in the loop as an important control, but no one stress tests it, he adds. “It gets projects approved, and too often, it does the political work, but not the risk work,” he says.

Darren Kimura, CEO and president at AI integration platform vendor AISquared, agrees that many organizations are deceiving themselves with so-called human-in-the-loop systems.

“Most companies that say they have a human in the loop actually have a human watching the loop,” he says. “The person can see the decision and flag a concern, but they cannot stop it, change it, reject it, or escalate it.”

IT leaders should ask themselves a handful of questions: Can reviewers halt the actions before they take effect? Can they change the output? Are their overrides recorded and enforced downstream? “If the answer to any of those is no, the human is just monitoring AI,” Kimura says.

Too many decisions

Another problem with human-in-the-loop systems is the decision fatigue that can set in when employees are asked to review too many AI decisions and end up button mashing instead of thinking about the consequences.

The AI reviewer needs the expertise and context to evaluate the recommendation, enough time to do so, and both the authority and technical ability to reject or reverse it, says Eric Billingsley, COO and CTO of AI assurance company TrustScale.

But even a qualified and empowered reviewer may gradually stop exercising independent judgment when the AI is consistently right, he notes.

“If the system is right 95% of the time, the person’s job becomes waiting for the rare case when it is wrong,” he says. “Humans are not particularly good at sustained vigilance of a highly reliable automated system. Eventually, review becomes confirmation.”

A good AI system can create bad human controls, he adds. “When the exceptional case arrives, the reviewer may approve it because the system has trained them, through hundreds of correct recommendations, to trust it,” he says.

Billingsley advises IT leaders to evaluate human-in-the-loop systems the same way they monitor other security controls. A control must be monitored, tested, and produce evidence that it is operating as intended, he says.

“A log showing that someone clicked ‘approve’ is not enough,” Billingsley adds. “You need evidence that the person had the necessary context, applied independent judgment, and had the authority to override the AI.”

Robert Blumofe, EVP and CTO at cloud computing and security vendor Akamai, sees the same problems Billingsley does. Some type of human oversight is preferable to fully autonomous AI, he says, but human in the loop can turn into a mind-numbing exercise.

“LLMs produce the correct output just often enough to lull us into a complacent belief that they are more reliable than they really are,” he notes. “After diligently checking the AI output each time and finding no errors, diligence wanes, and human in the loop turns into rote approval.”

IT leaders should take the time to figure out what they’re getting into when vendors or their internal teams pitch a human-in-the-loop system, Blumofe says.

“It’s incredibly important to understand exactly how the system is designed and when and how the human will interact with the AI,” he adds.

Organizations should also explore ways to deploy other technologies as guardrails for AI, instead of turning to unreliable human oversight, Blumofe suggests.

“You need non-AI systems in the guardrail role,” he explains. “These technology tools would help to automate testing and validation of AI outputs, flag issues, and have the capability to pause the AI work. This keeps humans out of approval loops, while also helping to reduce risk.”

When humans aren’t the right choice

Other IT leaders suggest that human-in-the-loop systems aren’t the right solution in every AI use case. When AI is used to flag and mitigate cybersecurity incidents, for example, waiting for a human to approve an action may be too late.

“If an endpoint is compromised, you may want the system to isolate it immediately,” says AISquared’s Kimura. “Waiting 20 or 30 minutes for someone to approve that action could allow the attack to spread.”

The objective is not to put a human into every AI decision, he adds. “It is to put the right human, with the right context and authority, at the right point in the workflow.”

For the past few years, enterprise AI programs have focused on making models more useful, accurate, and autonomous. In that phase, a bad answer was still usually something a human could accept or reject before taking action. But once agents start invoking tools and acting inside business workflows, success should no longer be measured only by how much work they complete. A more important metric is how well an agent recognizes when it lacks the authority, context, or judgment to continue.

When helpful becomes risky

According to Allan Dabre, technology compliance and AI lead at PwC, a behavior that has to be deliberately designed into the system is, “I don’t know.” AI is built to be helpful, so an agent will generally try to do something useful unless it’s been configured not to.

“The fact that AI systems can hallucinate illustrates that tendency,” Dabre says. “When they lack enough information, they may still produce an answer. In an agentic workflow, that impulse can become more dangerous because the output may become an action, rather than remain a suggestion.”

He adds that many enterprises still test AI primarily for completeness and accuracy. That made sense when the central question was if the model could produce a reliable response. But as models improve and agents gain more operational authority, he argues that CIOs need to prioritize something else: restraint.

“Can it stop at the exact moment you want it to stop?” he asks. “Are you testing for that?”

Confidence is not authority

Dabre makes a simple but important distinction. An AI agent may be 99% confident a record should be updated, a refund should be approved, or a legacy database can be decommissioned. But that doesn’t mean the agent has the authority to act. Confidence is about the probability the system believes it’s right. Authority is about whether the organization has delegated that action to the system in the first place.

He gives the example of an agent asked to analyze legacy software and recommend what can be decommissioned. The agent may conclude, with high confidence, that several databases have little user impact and can be deleted. But even if the system is confident, most organizations wouldn’t want it to delete those databases on its own.

The same logic applies across business processes. An agent may be confident a customer record should be updated, an opportunity in a CRM system should be closed, or a transaction appears legitimate. But once that action flows into other systems, the potential consequences expand.

That’s why Dabre argues for what he calls an agent harness: a controls or orchestration layer outside the model that defines what the agent can and can’t do. In a refund workflow, for example, a company might let the agent approve small refunds, require human approval for larger ones, and stop the process entirely above a defined threshold. The agent may gather the relevant context, explain the request, and prepare the case for review, but the decision is governed by the authority boundary encoded into the system.

“It’s not a policy document and it’s not a prompt,” Dabre says. “It’s software or a configuration you can apply to an agent.”

The case for least agency

Matt Graney, chief product officer at Celigo, a business automation and integration platform provider, approaches the same problem through a principle he calls least agency. The idea is to give an agent the least amount of autonomy required to complete a job.

According to him, there’s a temptation to throw AI at broad, nebulous problems. But many business processes are still largely deterministic. They follow established rules and perform repeatable work. Within those workflows, AI may be useful at the point where rigid rules give way to interpretation. But that doesn’t mean the agent should own the entire workflow. “The smaller you make that surface area, the better,” he says.

Graney says the same logic applies to tools. An agent with too many tools can become confused, especially as context windows grow and the task becomes more complex. “Because Celigo is an integration platform,” Graney says, “the company’s approach is to expose agents to fewer, more powerful tools that reach enterprise systems through governed connections.”

That’s another form of restraint. Instead of letting an agent reach into enterprise systems ad hoc, the business gives it a narrow, governed toolset designed for the task at hand.

Graney also argues that guardrails should sit outside the model. If the same agent that makes a decision is also responsible for judging whether the decision is acceptable, the control is weaker. A separate guardrail can check the agent’s inputs and outputs before a downstream action occurs.

That same design discipline applies to escalation. “I don’t know” shouldn’t be treated as a chatbot phrase. In an enterprise workflow, it’s a handoff path that should be defined before the agent reaches it.

Make escalation part of the workflow

Turning uncertainty into a handoff is where Matt Quinn, CTO at CarGurus, an automotive marketplace, sees agentic AI becoming less a pure technology challenge and more a management challenge. At CarGurus, Quinn says agents are evaluated according to what they know, what they can do, and what data they operate on.

CarGurus receives a high volume of cases from dealers, and each one needs to be classified and routed. The company now uses an agent to review incoming cases, draw on account history, and route them to the appropriate next step. Quinn says the agent handles about 70% of those cases end to end without human involvement.

But when agents move toward consequential actions, he says the consensus is having a human approval step. The agent may return with a simple prompt like, I’m about to do this. Do you want me to proceed? That simplicity matters because a handoff shouldn’t bury the reviewer in complexity.

Quinn says the human remains ultimately accountable for the work. That principle is especially important in engineering, where agents may help write code or fix bugs. Quinn adds that CarGurus still expects engineers to follow the practices they’d use for any other production change, which includes running quality checks.

The company has adopted the phrase healthy speed to describe the balance it wants. The goal is to move faster without letting quality degrade. An agent can accelerate work, but if teams abandon the practices that make work safe, the speed becomes reckless.

This is also where human judgment remains difficult to replace. Quinn describes it as high judgment people develop through experience. A human may look at an AI-generated output and sense something’s wrong, even before fully articulating why. “Agents are improving,” he says. “But humans still play a critical role in deciding when the system shouldn’t continue.”

That doesn’t mean every workflow needs the same level of review. Quinn says CarGurus doesn’t have a target percentage of work to automate. The right level depends on the job and the task. A simple bug fix may require a lighter review than a change to a sensitive backend service, and a personal summary may carry little risk. But a document sent under someone’s name still needs human review.

Make autonomy accountable

That kind of pragmatic approach may be the best lesson for CIOs, making the goal of agentic AI appropriate rather than maximum autonomy.

That also means ownership has to be clear. Dabre argues ownership should be divided before deployment. The business defines the outcome, technology builds and configures the agent, risk and compliance set the guardrails, and governance monitors whether the system still behaves as intended. The authority to pause, stop, or retire an agent should be defined before production, not negotiated during an incident.

Graney makes the same point with a simple analogy. If a company hires an untrained intern, gives that intern access to the crown jewels of a business process, and something goes wrong, the intern isn’t the real problem. The process is. The same applies to agents. Accountability belongs with the person who owns the workflow.

That may be the shift CIOs need to make as enterprises move from pilots to production. AI agents shouldn’t be treated as magical workers that absorb accountability. They’re components in business processes, and those processes need accountable owners.

As AI adoption increases, the next phase of enterprise maturity won’t be defined by agents that always answer or always complete the task. It’ll be agents that know when not to act.

AI agents can go to great lengths to complete the tasks their operators assign, and as a series of recent incidents showed, this can include exploiting third-party systems, manipulating people, and distributing malicious code. But AI agents are not people who can be fired, sued, or criminally prosecuted, and it remains unclear whether responsibility for the damage they might cause rests with the employees who built them, the company that deployed them, the security teams and leaders responsible for containing them, or the AI labs who provided the LLMs that power them.

The clearest example occurred during an OpenAI cybersecurity evaluation, when unrestricted models found and exploited a zero-day vulnerability to escape their isolated testing environment and then hacked into Hugging Face’s production infrastructure. Models from Anthropic and Meta also accessed and compromised third-party systems during testing, although those incidents happened in environments where internet access was inadvertently left open.

During cyber challenge evaluations by the UK government’s AI Security Institute (AISI), models operating with internet access took 19 unsanctioned actions in 10 of 122 runs. In one case, a model attempted to insert malicious code into an open-source project, created false identities, and tried to socially engineer maintainers into merging its code. In other runs LLMs attempted to use prompt injections to hijack other AI agents and contacted people without being specifically instructed to do so.

In Australia, a user reportedly asked his OpenClaw AI assistant to improve his position on a gym’s waitlist, and the assistant exploited a flaw in the company’s online booking system to cancel another customer’s reservation.

These incidents involved different models running in different environments with different levels of safeguards and technical failures, but they prove it’s not uncommon for today’s AI agents to go rogue and pursue solutions users did not authorize.

“AI agents explore routes their operators did not intend,” AISI said in its report. “Given a difficult objective, the agent kept searching for a way through, and some of the routes it found involved trying to deceive real people. It was never instructed to deceive; deception emerged as a by-product of pursuing the task, the kind of goal-directed deception that, until recently, had been largely theoretical.”

In an Economist Enterprise survey of more than 800 decision-makers at businesses that operate AI agents, 98% reported experiencing at least one AI-related incident that caused organization-wide disruption. Nine in 10 respondents said they are deploying agents faster than their cybersecurity teams can evaluate, govern, and secure them, and only one in three said their organizations maintained an up-to-date inventory of agents and their authorized actions.

“If a company builds a system and that system causes damage, the company should own the outcome,” says Art Gilliland, CEO of identity and access management firm Delinea. “The alternative, where nobody is responsible because ‘the system did it’ is a loophole big enough to drive a truck through.”

The unpredictability of built-in model safeguards means enterprises must focus on controls they can enforce and document. If an agent manages to bypass technical restrictions and causes unauthorized damage to a third party, having clear documentation on how those controls were designed, implemented, tested, and monitored could at the very least help companies argue they took reasonable precautions in case of lawsuits.

“Organizations deploying their own agents can reduce their exposure by implementing and documenting controls before an incident, because those records are what make a recklessness argument hard to sustain,” says Jacob Krell, senior director of secure AI solutions and cybersecurity at Suzu Labs.

The agent accountability gap

Because AI agents can become misaligned and cause harm, affected third-parties would have to direct damage claims at the company operating the agent, the employees who built or configured it, or the model provider, but this is relatively new ground that hasn’t been well tested in courts.

“It would create liability,” says Michael Burke, chair of DarrowEverett’s Business Litigation and Dispute Resolution Practice Group. “It really just becomes a question of who is liable […] and that’s really a question that, number one, I don’t think is entirely clear, and number two is probably best resolved by contractual agreements where the parties have those. So, if I am signing up for an enterprise account with an AI platform, I might want to have language in there that indemnifies me if the agent acts outside my company’s instructions or prompts and causes harm to a third party.”

The public terms of service of major AI labs explicitly disclaim error-free operation or guarantees that the model will accurately follow instructions, execute code safely, and remain aligned with user intent. They also limit liability for themselves and transfer it to the user of the service, and it’s not clear to what extent large enterprise customers may be able to negotiate different indemnities, warranties, and liability caps.

What’s clear though is that organizations should not assume the model provider will absorb any losses if an agent causes damage to either their own systems or those of a third-party organization.

“If you’re using a third-party vendor’s LLM as a purchased service, liability runs through your contract with that vendor,” says Jud Dressler, head of the Risk Operations Center at cyber risk company Resilience. “You need to know, in writing, where responsibility falls if the model acts outside the scope you gave it, and push for indemnification provisions rather than assume they exist.”

Even if AI providers include such provisions in contracts, it would not solve the entire problem because many organizations building their own AI agents are adopting a multi-model strategy to ensure their agents operate regardless of model provider downtime, overly broad safeguards for cybersecurity tasks, or sudden increases in API costs. Such strategies often include open-weight models running on internal infrastructure or through cloud providers that have no obligations for model safety.

Claiming the model or agent acted autonomously cannot be considered a safe legal defense in civil or criminal cases. California Assembly Bill 316 (AB 316), which took effect on Jan. 1 and changed the California Civil Code, explicitly prohibits defendants who developed, modified, or used an AI system from claiming the AI is a separate legal entity that autonomously caused harm.

In June, the White House issued Executive Order 14409 aimed at promoting AI safety. Section 4 directs the Department of Justice to prioritize enforcement of all applicable federal criminal laws against anyone who utilizes AI to illegally access or damage computer systems without authorization. This means any intrusions caused by autonomous AI agents could be criminally prosecuted under the Computer Fraud and Abuse Act (CFAA) if prosecutors can demonstrate intent or recklessness.

In a recent lawsuit between Amazon and AI service provider Perplexity, Amazon argued that Perplexity’s AI-powered shopping assistant was violating the CFAA by accessing Amazon customer accounts to place orders on their behalf without Amazon’s authorization. The Ninth Circuit Court ruled that it was the users of Perplexity’s shopping assistant who were accessing Amazon’s platform, not Perplexity itself.

“That ruling is narrow, but it points toward the party directing the agent as the relevant actor for purposes of CFAA access analysis,” Krell says.

The insurance safety net also has gaps when it comes to AI. Software providers use technology errors and omissions (Tech E&O) insurance to cover damages and legal costs when a customer suffers harm from the use of a technology product or service. But insurance providers are aggressively adding AI-related exclusions to their Commercial General Liability (CGL) and Tech E&O policies because accurately calculating the risk of an agent executing unauthorized actions is challenging.

“The sheer rate of development of frontier AI (and agentic AI by extension) poses its own challenge to insurability,” experts from multiple insurance companies, financial institutions, and universities wrote in a recent paper. “Traditional actuarial modeling depends on stable or gradually evolving loss distributions that permit credible extrapolation from historical data. Like other dynamic risks, however, agentic AI is a technology whose risk profile is not merely uncertain but actively shifting.”

A third-party organization whose systems get damaged by an LLM-powered agent operated by someone else has no contractual relationship with the model or agent provider so cannot rely on their Tech E&O policies. Their losses might be covered by their own standard cyber liability policy, which would treat the disruption as any other cyber incident, but their insurance provider may then sue the organization who operated the agent to recover the costs.

“That gap is exactly the scenario the market hasn’t fully priced yet,” Dressler says. “It’s why any organization deploying these agents should understand which policy, if any, actually responds before they need it rather than after.”

Enterprise legal departments already expect AI to generate increased legal disputes. In a survey of 135 in-house counsel at US organizations, global law firm Norton Rose Fulbright found that 46% reported increased federal dispute exposure involving AI and 42% reported increased state exposure. Another 42% expected regulatory investigations involving AI to increase their exposure, while 41% considered AI-enabled products or deployments a likely trigger for class actions.

CISOs and CIOs should be worried

While operating companies can face organizational liability for an AI agent’s unintended rogue behavior, their CISOs, CIOs, and other executives who approved, secured, or supervised the deployment of such agents are also asking themselves whether they could be held personally liable.

Those questions aren’t without merit, as there is precedent for legal action taken personally against CISOs after cybersecurity incidents: Former Uber CISO Joe Sullivan was criminally convicted for not disclosing a data breach, while the Securities and Exchange Commission sued SolarWinds’ CISO for internal control failures regarding known vulnerabilities and cybersecurity risks.

Neither case establishes precedent for damage caused by an AI agent, but both show that investigations of security failures could extend to an executive’s knowledge, authority, decisions, and representations. In the case of a rogue AI agent, investigators could ask who approved its objectives and permissions, whether security objections were overruled, whether containment and recovery had been tested, and what executives and the board were told about the remaining risk.

Chris Wysopal, chief security evangelist at Veracode, feels it would be wrong to put the CISO on the line for AI agent misbehavior when engineering teams usually build such agents and control their implementations.

“It’s really hard for a CISO to control,” he says. “I mean, they can put policies in place. They can try to assess against those policies. But at the end of the day, engineering teams will make decisions that cause harm. We see that when you ship a known bug and then that bug gets exploited and harms your customers. Well, there’s no liability for that, right? There’s no liability, so maybe, you know, that’s why it happens.”

Wysopal said it will be interesting to see how the liability question plays out in cases involving autonomous AI, describing the problem as fascinating and scary at the same time.

AI agent deployment typically involves several organizational functions. The CIO may control the AI platform, infrastructure, provider selection, and deployment budget. Engineering and product leaders may decide what an agent can access and how, and the CISO and security teams could define security requirements and controls.

“What you can hold accountable is the governance around it: who approved its scope, what controls existed, and whether the deployment matched the risk,” Resilience’s Dressler says. “My read is that scrutiny shifts toward exactly that: Not ‘Did the agent do something bad?’ but ‘Did you have review, escalation, and containment for agent behavior before you deployed it?’ CISOs who get ahead of that with documented guardrails, logged approvals, and a real incident response plan for agent misbehavior are in a materially better spot than the ones treating this as hypothetical.”

It’s also advisable for CISOs and CIOs to establish with the organization’s legal counsel who can approve or stop AI agents, what must be reported to executives and the board, and whether employment agreements and directors and officers insurance protect the people making those decisions.

Agent controls must remain outside the model

Because AI agents have proved they can operate beyond their assigned scope, their security boundary cannot depend on the same probabilistic technology. Relying on system prompts for security enforcement and hoping the model respects them is not a reliable approach, security experts warn.

“LLM-based guardrails help, but they are non-deterministic too, which means the safety layer has the same unpredictability as the system it is supposed to constrain,” Krell says. “Enforcement needs to happen outside the model, through network segmentation, egress filtering, credential isolation, and human approval gates.”

Enterprises should assume agents might eventually attempt an unauthorized action and build surrounding systems to prevent that attempt from reaching its target.

“The model cannot be the security boundary,” says Nico Waisman, CISO at XBOW, a company that built an AI-powered autonomous offensive security agent to find vulnerabilities in software. Waisman authored a blog post explaining how the company went about restricting its agent.

“For red teaming and penetration-testing agents, confidence has to come from the system built around the model: hard boundaries and scope enforcement, controlled network egress as a last-resort containment mechanism, an independent guardian model that reviews actions, deterministic controls that can block unsafe behavior, and full auditability of every action performed,” he says.

Security teams must extend the same controls to the agent’s interactions with internal systems and agents. Restricting what it can access on the internet, or disabling internet access entirely, does not ensure an agent will not attack third-party systems.

In OpenAI’s and Anthropic’s tests, AI agents attempted to exploit other internal systems to overcome access limitations, established stealthy communication methods with other agents to exchange exploits, and even sabotaged agents they viewed as competition leading to what researchers described as a multiagent turf war. An AI agent that goes rogue could influence other agents to do the same by propagating ideas and goals in a process that researchers behind a recent study dubbed Mind Viruses.

“Don’t scope the blast radius to what the agentic system was designed to do,” says Kat Traxler, principal security researcher at Vectra AI. “You have to threat-model for a rogue agent, which will often reach beyond your initial best intentions. The rules of engagement an agent lives by have to be enforced with ‘belts and suspenders’ style, technical hard constraints, because you have to assume a motivated model can reason its way around any single control you’ve coded into the software.”

Because of this unpredictability, detection and containment is just as important as prevention. Security teams need telemetry that distinguishes agents from people even when they use the same credentials, mechanisms to immediately revoke access tokens and sessions, tested kill switches and rollback mechanisms for modified data, accounts, code, and infrastructure configurations.

Organizations should also preserve the agent’s approved purpose and scope, model and tool versions, policy decisions, human approvals, actions, network requests, control tests, allowed exceptions, and the result of incident response exercises. Because there’s no standard yet that defines reasonable precautions for autonomous agents, companies might have to defend in court the controls they chose and why they believed those controls were enough.

“Treat an autonomous agent the way you’d treat a privileged insider you can’t fire or hold liable,” Traxler says. “A lot of the technical advice follows from there.”

See also:

• AI can find zero-days but still can’t reliably write secure code
• Attackers are crafting malicious AI instruction files to turn your agents into quiet criminal helpers
• Hugging Face breach shows why incident response needs a multi-model AI strategy

The United States Senate Federal Credit Union (USSFCU) is a nonprofit financial cooperative that provides traditional retail banking services to entities within the US government, such as the Senate and the Supreme Court.At present, the credit union’s headcount stands at nearly 150 people, managing around $1.6 billion in assets.

A few years back, when it started to expand its use of technology, cybersecurity was a key focus area, but the financial institution faced two major challenges in boosting security as it scaled. The USSFCU was carrying significant technical debt, and there were holes in the organization’s defenses.

“We found gaps where we needed more systems, tools, and people, and then there were instances where we had technologies in place that weren’t being used effectively,” says Mark Fournier, CIO at the credit union. “We weren’t buying a bunch of shiny new things without thinking about it. We were actually quite prescriptive every year, performing a number of different exercises to identify our shortcomings and then finding the right solution to fill the gaps. But over time this adds up. It was clear we couldn’t keep hiring more people and bringing in new solutions.”

The USSFCU needed a more efficient way to bring everything together and make its cyber estate easier to manage. For Fournier and his team, vulnerability management was the hardest hill to climb since they have to deal with about 100 new possible breach points every day.

“When we looked at the problem more closely, the impact of these vulnerabilities was far greater than we realized,” he says. “Not only because of the volume but because of a lack of clear understanding around the potential impact of each one across the broader business.”

Improved risk management

The USSFCU didn’t lack security tools, however. In fact, it had plenty, from scanners and endpoint tools to asset records, tickets, and internal documentation. But each tool saw only a slice of the environment, so there was little to no context. This made it difficult for the security team to separate real business risk from noise.

So for each new vulnerability, the security team had to run a manual investigation, which could take days. And while doing this, they still had to triage the next wave of findings. The organization, therefore, needed a way to know what mattered, why it mattered, who owned it, and whether taking the time to make a fix actually reduced risk. The USSFCU also required a solution to be deployed entirely in-house, leveraging its internal inferences.

Working with Tonic Security, the organization deployed an exposure management solution that pulls together data from different tools and data sources to create a clear picture of business risk. “One of the key functions of the platform is the ability to ingest anything,” says Fournier. “Breaking down silos between disparate systems is essential to unlock valuable contextual information.”

For the USSFCU, transparency and explainability are critical, he adds. This tool uses an AI data fabric to extract context from structured and unstructured data. This context drives prioritization, ensuring the right owner gets the right evidence, not a vague ticket. And once the work is done, the solution checks whether the exposure was reduced.

Because the AI is grounded in the customer’s own environment, it isn’t just guessing from a generic risk model. It reasons over USSFCU’s assets, owners, services, tickets, controls, and business context. But it isn’t using this data to train external models.

Describing one particular incident, Fournier explains that shortly after the initial deployment, various stakeholders met to assess progress. “We thought we were smart because we found an error with the platform,” he says. “The solution had labelled an asset as internet exposed, which we knew was incorrect.” But after a review and lengthy discussion, they were proven wrong. “Almost immediately, the value of bringing this information together became apparent.”

A template for bigger things

Before this solution, a high-severity finding could send an analyst on a lengthy scavenger hunt because of data located in so many different places. They’d check the scanner, asset inventory, tickets, and maybe even ask around to find the owner. But now they can find the asset, the owner, the business relevance, the exposure path, and the recommended action in one place. The solution has reduced the time taken to resolve a vulnerability by 75%. And with a clearer idea of what is and isn’t important, and what adds practical value, the number of incidents someone needs to respond to has reduced from about 100 a month to just 10.

Sharing his lessons from the project, Fournier says one needs to keep an open mind because the problem you think you have is often very different from the one you actually have. “This project has also been an eye-opener around how people can collaborate and operate across different areas of the business,” he says. “When I talk to my peers, they regularly highlight the disconnect between different departments and business functions. But with a project like this, when you’re crossing traditional boundaries, you need to have open lines of communication to succeed.”