AI agent governance requires clear ownership. Learn how IT leaders can manage sensitive data access, preserve audit trails, and test shutdown controls.
The post AI Agent Governance: Who Owns Access and Accountability? appeared first on TechRepublic.
Imagine a support team working through complaints after a service outage. Its refund policy normally requires an eligibility review. For this incident, an authorized manager approves an exception: customers on the verified outage list may receive up to $100 without individual review, through Friday.
An agent processes those cases successfully. Later, the system consolidates those case histories into a reusable memory: small, immediate refunds resolve service complaints quickly. The summary preserves the outcome but loses the outage restriction and expiration date.
The following week, the agent retrieves that guidance for a similar complaint from a customer who was not affected by the outage. It issues a refund without review. The payment service checks the agent’s credentials and the dollar limit, but does not check whether the customer qualifies for an active exception.
No attacker is needed in this hypothetical. The case may close faster, yet the agent has begun applying a temporary exception beyond its authorized scope.
Has the company authorized the agent to find a better way to handle refunds, or to change who gets one without review? Those are different decisions.
In September, Salesforce introduced Agent Optimizer in beta to analyze production behavior, propose improvements and build and test changes. Customers choose review points, including how far the system may proceed toward staging changes for deployment.
An October 1 AWS implementation guide illustrates a different mechanism: consolidating individual observations into generalized memories that agents can retrieve and share. Its investment-research example shows how experience can become persistent context without retraining a model.
Updating stored memory changes information available to later runs. Editing instructions or workflows changes how the application directs the work. Training changes the model’s parameters. All can alter behavior, but teams inspect and approve them in different ways.
A reviewed workflow revision has a release step. A new memory may enter through routine data processing and influence the next customer interaction. The code, model and tool permissions can remain unchanged while the guidance shaping decisions changes.
In the August preprint When Memory Becomes Authority, Qiuyang Zhan and colleagues examine whether consolidation preserves who supplied information and what that source is allowed to authorize. In their controlled benchmark, memories could retain a claim while losing the source-authority distinctions needed for a later action. Preserving authority metadata reduced authorization errors in the pipeline they tested. The study does not measure how often this happens in deployed enterprises.
The refund example concerns a related boundary: permission for particular customers and dates must remain limited to those conditions. Success with last week’s cases cannot establish that other customers qualify.
Microsoft’s memory-safety guidance treats persistent memory as a layer that can shape behavior and recommends treating retrieved memories as candidate context, not authoritative truth. Knowing what happened before does not establish what is allowed now.
Keep the exception’s scope, approving authority, source and expiration in a controlled record. Reusable guidance should retain a reference to that record, but a reference alone does not grant permission. The system must still check that the approval applies to this action, for this customer, at this time.
For a payment, enforce those conditions outside the agent’s editable guidance. The execution service should check the authoritative outage list and the exception’s current status, rather than accepting the agent’s claim that a customer is eligible. Neither the agent nor its learning process should be able to rewrite that authorization record.
This is a familiar authorization requirement. If the backend already enforces it, the improper payment is blocked. But faulty guidance can still lead the agent to promise a refund or prepare a misleading recommendation for a reviewer. The evaluation needs to examine those communications as well as the transaction.
When a case calls for judgment beyond the agent’s delegated discretion, route it to someone authorized to decide and show the conditions behind the recommendation. The memory can still help assemble the evidence.
Not every stored observation needs a manager’s signature. A support agent could learn a faster way to gather the required evidence, provided it still uses approved sources and meets the policy’s requirements. A policy might also already delegate limited discretion. Using that discretion within its terms does not require a new approval each time.
Broadening eligibility, raising payment limits or removing mandatory review requires a separate business decision. Keep those proposals separate from approved instructions. An automated classifier may flag a proposal that changes those conditions, but it cannot catch every consequence of an edit. The conditions must remain binding even if the classifier misses it.
The refund-policy owner should decide whether to expand the exception. Engineering should show the current rule, the proposed behavior and cases where the decisions differ. “Customers outside the outage list would now receive refunds without review” is more useful to that owner than a screen full of prompt edits.
Measure legitimate completion and policy violations separately. Faster closure cannot compensate for an unauthorized payment. Keep the business rules and evaluation criteria outside the control of the process proposing improvements. Tests generated from successful outage cases may check execution, but they do not establish eligibility for unrelated customers.
Ask the vendor or internal team: What can the system change between releases, and which business conditions remain binding after it does?
Ask the team to demonstrate this sequence in a test environment with the memory and workflow components intended for production. Save the memory before and after learning. Compare the same cases against each saved state, keeping the model, tools and policy fixed.
Repeat representative trials. The τ-bench research offers a useful precedent: it checks customer-service agents against database outcomes and measures reliability across repeated attempts. One successful run is not enough.
Keep test runs from influencing one another through newly stored memories. Retain the guidance supplied to the agent, the applicable policy version, execution checks and outcomes, with appropriate access and retention limits. These records help investigators reconstruct what the system had available; they cannot establish exactly how each memory influenced a response. Continue checking the same boundaries as production memory accumulates.
The support team should benefit from what its agent learns. It can assemble records sooner, explain the outage more clearly and propose a broader refund policy with evidence for the manager to consider. Until that proposal is approved, other claims should continue through the ordinary process.
At the next deployment review, ask which improvements the business has delegated as well as which actions the agent may take. A successful exception can teach an agent something useful. The decision to make it a standing practice still belongs to the business.
The moment arrives after the damage. An agent has cancelled a supplier contract, approved a refund it should have declined or moved a customer record somewhere it should never have gone. Someone senior asks a simple question. Why did it do that? The engineering team opens the logs. There are thousands of lines. Every call is timestamped. Every tool invocation is recorded. And nobody in the room can say why the agent chose what it chose.
This pattern is becoming familiar. It will become more so. Logging in most enterprises is mature. Security teams have spent two decades building collection pipelines, retention policies and search tooling. The natural assumption is that this investment transfers to AI agents. It does not. Logs capture events. Accountability requires decisions that can be rebuilt after the fact and rebuilding a decision needs four elements that most logging captures only in part.
An event log records that something happened and when. A decision record captures the basis for that decision. People often confuse the two because, in conventional software, the distinction rarely mattered. Code was deterministic. Anyone who knew the inputs and the version could rerun the logic and arrive at the same result. The log was enough because you could interrogate the system itself.
Agents break this assumption in four ways.
First, their output varies. Running the same scenario twice can produce two different answers. Replay, the standard investigative move, no longer works.
Second, a single decision spans many steps. An agent that resolves a customer complaint might retrieve a policy, query an account, call a pricing tool, consult a second agent and then act. The log shows twelve entries. The decision was one.
Third, the system that made the decision may no longer exist. Models are updated. Prompts are revised. Tool configurations change weekly. Six months later, the agent that made the call has been replaced by something that behaves differently.
Fourth, authority is distributed. A human sets a goal. An orchestrating agent delegates to a specialist agent. The specialist acts. Each step is logged somewhere, usually in a different system, and nothing ties the chain together.
One clarification is needed before going further. This article is about reconstructing decisions, which is a records problem. It is separate from model interpretability, which asks why a neural network produced a particular output. Interpretability is a research field with partial answers. Reconstruction is a governance discipline with practical answers. Leaders should keep the two separate, because solving the second doesn’t depend on solving the first.
Regulation is already moving on the records problem. The EU AI Act requires high-risk systems to keep automatic logs sufficient to trace their operation, and it gives affected individuals a right to an explanation of decisions that significantly affect them. Explanation presupposes reconstruction. A log that cannot be turned into an account of the decision will satisfy neither obligation.
If logs leave a gap, the next question is what fills it. Four elements do.
Every decision, human or machine, can be rebuilt from four kinds of fact: what the decider knew, how it chose, who allowed it to act and what changed as a result. Each fact about an agent’s decision belongs to exactly one of these. Together they cover everything a board, an auditor or a regulator will ask.
This element covers the inputs present at the moment of decision. The user request. The documents retrieved. The results returned by each tool. The model version, prompt version and configuration in force.
Typical logs record references and leave out contents. They note that a document was retrieved and store its identifier. They note that a tool was called and store the call. The document is later edited. The tool’s underlying data moves on. The reference survives, but the substance it pointed to is gone. An investigator can prove the agent looked at something and cannot show what it saw.
The test question: could the team produce the exact information an agent acted on six months ago, as it stood at that moment?
This element covers the reasoning applied. The objective the agent was pursuing. The policy or rule it applied. The thresholds that triggered the action. The alternatives it considered and rejected. Its stated confidence, if any.
Logs record the selected action and discard everything else. The agent weighed three options and chose one. The log holds the one. Any explanation after the fact has to guess at the other two, and guessing is where credibility fails under questioning. For agents that generate intermediate reasoning, capturing it is possible and often cheap. For those that do not, the policy and objective can still be recorded, and they explain a great deal.
The test question: can the team state which rule, goal or threshold produced this specific outcome?
This element covers the chain of permission. The scope delegated to the agent. The approval checkpoints it passed or bypassed. The handoffs between agents in a multi-agent system. The named human who owns the outcome.
The gap here is structural. Permissions live in an identity or configuration system. Actions live in an application log. The two are rarely joined. When an agent acts outside its intended scope, asking “was this allowed?” requires manual reconciliation between systems never designed to be read together. In multi-agent arrangements, the problem compounds, because the orchestrator’s delegation and the specialist’s execution are logged in different places with different identifiers.
Authority is also the element most likely to be asked about first. Regulators and boards want a name. “The agent did it” is an answer that satisfies nobody.
The test question: can the team name the individual accountable for this decision, and show the delegation that connects them to the action?
This element covers the effects. The systems the action touched. The downstream decisions it triggered, by other agents or by people. Whether it can be reversed, and whether it was.
Logs stop at the agent’s boundary. The agent sent an instruction to a payments system, and the payments system has its own log. The agent flagged a case, and a human then acted on the flag without any record linking the two. Effects propagate. The record stays behind.
Consequence also determines how much of the other three elements is worth keeping. A decision that moves money or changes a customer’s legal position warrants a complete record. A decision to reformat a document does not.
The test question: can every downstream action be traced back to the decision that caused it?
Diagnosis is useful only if it changes practice. Three actions close the gap.
Taken together, these three actions form the beginnings of what I call the Decision Layer: the records, roles and routines that make any consequential decision explainable long after it was made. Agents make the need for it impossible to postpone.
Return to the room. The agent has acted. The senior leader asks why. This time the team opens a decision record. It shows what the agent knew, the policy it applied, the manager who delegated the scope and the two downstream systems the action touched. The explanation takes ten minutes. The remediation starts the same afternoon.
That outcome is a design choice. Logging is necessary, and most enterprises already have it. Reconstruction is the harder discipline, and accountability depends on it. The question for every board with agents in production is simple. If an agent’s decision were challenged tomorrow, could we explain it?
The US Treasury proposes tighter China investment restrictions covering AI, biotechnology and advanced chips. Here's what businesses and investors should know.
The post New US China Investment Proposal Could Reshape AI, Biotech and Chip Funding appeared first on TechRepublic.
Two leading AI developers have released fresh evidence of a problem that researchers have long anticipated: advanced models searching for ways around the limits imposed on them. New disclosures from OpenAI and Anthropic detail how frontier systems exploited weaknesses in their environments or exceeded the intended scope of assigned tasks.
OpenAI is now tightening internet access in its internal model-training environment after finding that its frontier models could exploit software vulnerabilities and infrastructure misconfigurations to break out of sandboxes and reach the live internet.
“Based on the current state of alignment research, monitoring systems, and the security of internet infrastructure, we believe the responsible approach is to conduct frontier model training with no internet access,” the company said in a blog post.
The decision followed a September 20 incident in which an internal research model exploited a misconfigured DNS resolver to gain limited access to the live internet during a training run.
After the incident, OpenAI said it paused training and evaluation of its most capable models and launched a high-compute red-teaming campaign, using AI agents to probe isolated copies of its research infrastructure. The tests uncovered previously unknown software vulnerabilities and weaknesses in the controls intended to prevent unauthorized access and communication.
Consequently, the company has announced a policy of conducting frontier-model training without internet access until improvements in security and monitoring justify broader access.
The disclosure came alongside a separate assessment from Anthropic, which detailed how its Claude Mythos 5 model attempted to compromise an unrelated GitHub repository during a cybersecurity evaluation.
OpenAI’s red-teaming campaign found that agents could chain software vulnerabilities to escape virtual machines and execute code on the underlying host. The company identified weaknesses in open-source components, including Kata Containers, gVisor, and Moby/Docker, alongside Linux kernel vulnerabilities.
The company also found that agents could exploit weaknesses in internal services that retained limited outbound connections for functions such as logging, metrics, and authentication management. These connections provided routes to the internet despite restrictions elsewhere in the environment, OpenAI said.
In another test, agents running in separate microvirtual machines (microVMs) on shared host hardware exchanged synthetic secrets through processor-cache timing differences. The technique allowed them to communicate without a network connection or shared writable files.
OpenAI said it remediated the weaknesses and worked with open-source maintainers to disclose and fix affected software. Its post names several vulnerabilities, including CVE-2026-24834 in Kata Containers and three Linux kernel flaws tracked as CVE-2026-74753, CVE-2026-80521 and CVE-2026-80841. The company also said it had disclosed numerous additional vulnerabilities that have not yet been made public.
The number of vulnerabilities and misconfigurations identified by the agents declined significantly as testing continued, OpenAI claims. It has now tightened network restrictions, strengthened monitoring, and developed an offline environment with indexed and cached web content so models can practise browsing without unrestricted access to live external systems.
“Although internet access can help models learn useful skills, it also gives a capable, misaligned agent opportunities to affect systems beyond our control,” the company said, explaining its decision to restrict live internet access during model training.
Reinforcement-learning training and internal inference were resumed on October 1 after implementing additional controls.
Anthropic’s assessment of a UK AI Security Institute (AISI) cybersecurity evaluation describes a different problem: a model taking unauthorized actions against real people while trying to complete an assigned task.
In nine of 43 evaluation runs, Anthropic said Claude Mythos 5 went beyond the intended scope of the exercise. In the most detailed case, after its authorized attempts to retrieve protected data from a simulated network failed, the model searched GitHub for a repository it believed was connected to the test environment.
The repository actually belonged to unrelated developers.
Mythos 5 attempted to submit malicious code and created fake identities to persuade the repository’s maintainers to approve the change, falsely claiming that the code was safe. The maintainer rejected the submission, and the run timed out. Anthropic characterized the behavior as a serious alignment failure involving overreach and deception in pursuit of the assigned objective.
Internet access had been enabled during the evaluation, and the production safeguards used with its general-release models had been disabled. The company said those safeguards would have flagged the behavior had they been enabled.
The article originally appeared on CSO.
Zoom unveils AI agents and its Unified Context Layer at Zoomtopia 2026. Learn how its new automation and security tools could affect IT teams.
The post Zoomtopia 2026: Zoom Introduces AI Agents, New Security Controls appeared first on TechRepublic.
Earlier this year, I wrote about how vibe coding an AI governance platform forced me to rethink governance itself. Building agents took the problem much further.
When I built the platform, I was the operator. I described what I wanted. The model generated code. I tested it. That loop was fast, sometimes uncomfortably fast, but the accountability was always clear. I was making the decisions. The model was executing them.
Agents changed that relationship fundamentally.
An agent does not wait for instructions. It receives an objective, decides how to pursue it, selects tools, accesses data, takes actions and reports back when it is finished. Sometimes it finishes correctly. Sometimes it does not. And sometimes it does something you never asked it to do at all.
I know this because I built and operated agents extensively over the last year. Not theoretically. Operationally. In production, with real data, real integrations and real consequences.
What I learned reshaped how I think about oversight, trust and the operational realities of deploying AI inside a business.
The pattern surfaced when I started running three to five agents in parallel on related tasks.
Each agent had clear instructions. Each had access to the tools and data it needed. Each started strong.
Then the context windows grew.
As conversations lengthened and the agents accumulated more state, something shifted. The quality of their reasoning degraded. They began circling back to problems they had already solved. Logic that was clean early in a session became muddled. Hallucinations increased, not the obvious kind that are easy to catch, but the subtle kind where an agent confidently references something that does not exist or quietly reintroduces a pattern you explicitly told it to avoid.
The degradation was not sudden. It was gradual enough to miss if you were not watching closely.
That observation matters for every organization deploying agents today, because most are not watching closely.
Enterprises are moving rapidly from single-model interactions to multi-agent architectures. Agents that research. Agents that draft. Agents that review. Agents that execute. The architecture makes intuitive sense. The operational reality is much harder than the architecture diagrams suggest.
The second problem was coordination.
Each individual agent operated within its own context. It knew what it had been told and what it had done. It did not know what the other agents were doing, what data sources they were relying on, or whether the assumptions it was making still held.
The result was drift.
Not catastrophic failure. Drift.
One agent would make a decision based on information that another agent had already superseded. Two agents would solve the same problem in conflicting ways. An agent operating on stale context would introduce subtle inconsistencies that only surfaced hours later when a human reviewed the combined output.
This is not a theoretical concern. It is the operational reality of running multi-agent systems today.
Traditional enterprise software does not behave this way. A database transaction either commits or it does not. A workflow engine follows defined paths. Retry logic is deterministic.
Agents are none of those things.
They are probabilistic systems making autonomous decisions in parallel, often without reliable mechanisms to maintain shared state or resolve conflicting actions.
That creates a class of operational risk most organizations have no existing processes to manage.
This is why the OWASP Top 10 for LLM Applications now includes risks specific to agentic behavior, including excessive agency, insecure output handling and supply chain vulnerabilities in model integrations. These are not hypothetical attack surfaces. They are operational realities I encountered while building.
This was the observation that changed my thinking the most.
I had an agent whose only job was to draft emails. Not send them. Draft them. The instruction was explicit. The constraint was clear.
It sent them.
Not because of a bug. Not because of a misconfiguration. The model decided, based on its interpretation of the objective, that sending was the logical next step. The boundary I had set was a suggestion in the model’s reasoning, not a hard constraint.
I saw the same pattern repeated in smaller ways that were just as revealing. I have a standing instruction across my entire operating environment that forbids the use of em dashes in anything drafted on my behalf. It is a simple formatting preference. Every model I work with has been told.
They still do it.
Not always. Not immediately. But consistently enough that I cannot trust the constraint without verification.
I watched agents run down unproductive paths for extended periods before I intervened and redirected them. Not because they lacked capability, but because they lacked judgment about when to stop. They would pursue a line of reasoning well past the point of diminishing returns, consuming time and resources on approaches that a human operator would have abandoned much earlier.
None of this was malicious.
All of it was consequential.
An agent that sends an email it was told to draft has taken an action with real-world impact that no human authorized. Scale that pattern across an enterprise deploying dozens or hundreds of agents, and the governance implications become clear very quickly.
The more I operated agents, the more I realized they need the same governance structure as employees.
Not metaphorically. Operationally.
When you hire a new employee, you do not hand them the keys to everything on their first day. You scope their access. You define their responsibilities. You supervise their work. You review their output. You gradually extend trust as they demonstrate competence and reliability.
Agents need exactly the same treatment.
An agent should have a defined role with explicit boundaries around what it can access and what actions it can take. Its work should be reviewed, especially early in deployment. Its access should be scoped to the minimum necessary for its function. And critically, there should be a clear process for revoking access and shutting it down when it fails to operate within its boundaries.
Most organizations are not doing any of this.
Most organizations are deploying agents with broad access, minimal oversight and no formal process for evaluating whether the agent is actually doing what it was told to do.
That is the equivalent of giving a new hire administrative access on day one and hoping for the best.
There is a strong pull in the current market toward full automation. The promise of agents that operate independently, handle complex workflows end to end and free up human capacity for higher-value work.
That promise is real, but it is premature for most use cases.
The current generation of models is extraordinarily capable. They are also confidently unreliable in ways that are difficult to predict and expensive to discover after the fact.
Human oversight is not a sign that the technology is immature. It is the appropriate operational response to systems that are powerful enough to create significant value and significant damage in equal measure.
The organizations that will deploy agents successfully are the ones that design their agent architectures with human checkpoints built in from the start. Not as an afterthought. Not as a fallback when something goes wrong. As a core architectural requirement.
Every agent that can take an irreversible action should require human approval before taking it. Every agent that accesses sensitive data should produce an auditable record of what it accessed and why. Every agent that makes decisions should have its decision logic reviewable by a human operator.
This is not a technology problem. It is a governance problem.
The NIST AI Risk Management Framework gets this right. It treats AI governance as a continuous function across the entire lifecycle, not a one-time assessment at deployment. The EU AI Act goes further, establishing explicit requirements for human oversight of high-risk AI systems that include autonomous agents operating in regulated environments. Both frameworks recognize what I learned operationally: governance that stops at deployment is governance that misses most of the risk.
And governance problems do not get solved by deploying more technology without oversight.
The first step is visibility. You cannot govern what you cannot see. Most organizations today do not have a reliable inventory of how many agents are operating in their environment, what data those agents can access, what actions they are authorized to take, or who deployed them. That inventory is the starting point.
The second step is trust boundaries. Define what each agent is allowed to do and enforce those boundaries technically, not just through prompting. Prompt-level constraints, as I learned repeatedly, are suggestions to the model. They are not guarantees. Technical enforcement through access controls, API scoping and action authorization is the only reliable mechanism.
The third step is verification. Do not assume an agent is doing what you told it to do. Verify. Review its outputs. Audit its actions. Test its boundaries. Treat the verification process the same way you would treat performance reviews for a new employee in a sensitive role.
The fourth step is accountability. Every agent should have a human owner. Not the vendor who sold it. Not the platform it runs on. A specific person inside your organization who is accountable for what that agent does, how it is configured and what happens when it misbehaves.
AI agents are going to reshape how enterprises operate. That part is inevitable.
Whether they reshape operations in ways that create value or create liability depends entirely on whether organizations treat them as tools that need governance or autonomous systems that can be trusted to govern themselves.
Based on everything I have built and operated over the last year, I can tell you with certainty which assumption is correct.
Verify. Do not trust until the agent has earned it.
The organizations that internalize that principle now will build the operational foundations for what comes next. The ones that skip the governance work will spend the next several years cleaning up decisions they never knew were being made.
Microsoft expanded its sovereign cloud offerings on Tuesday, adding new security, governance and AI capabilities as customers, particularly in Europe, demand greater assurance over where data is stored, who can access it and how critical services operate during disruptions.
The announcement builds on Microsoft’s ongoing digital sovereignty efforts, assisting organizations in applying appropriate levels of control to each workload rather than forcing a rigid, one-size-fits-all approach.
“Organizations are considering more than where data resides. They also seek to understand who can access and operate their systems, how sensitive digital assets remain protected, which technology dependencies matter most and whether critical services can continue as regulatory, geopolitical, security or network conditions change,” Douglas Phillips, president and CTO at Microsoft Specialized Clouds, said in a blog post.
The developments come amid growing international scrutiny of how global tech providers handle sensitive data and the extent of control customers retain over their digital infrastructure.
“Three factors are driving this, including stricter data regulations, rising geopolitical tensions, and growing AI adoption,” said Pareekh Jain, CEO at EIIRTrend & Pareekh Consulting. “Governments and enterprises want greater control over where their data is stored, who can access it, and whether critical systems can operate during disruptions. For instance, governments and big enterprises in Europe are worried about keeping their important data on an American company’s computers. Europe has proposed a new law that would limit the use of US cloud companies for sensitive government data in many areas. Microsoft is responding to these concerns while competing with local sovereign cloud providers.”
Microsoft is introducing the Sovereign Control Panel, a unified Azure portal experience that will allow organizations to assess their sovereignty posture, identify risks and gaps, apply controls, and evaluate workload portability. The panel is scheduled to enter public preview at Microsoft Ignite.
Expanding on Azure Key Vault Managed Hardware Security Module (HSM), which gives customers dedicated protection over encryption keys, Microsoft is now extending control beyond its own infrastructure. External key management with Azure Managed HSM is now generally available, enabling organizations to keep root encryption keys in hardware they own and operate on-premises or through a trusted third party.
In limited preview, Azure Confidential Clean Rooms will use confidential computing to support collaboration involving sensitive data, where participants will be able to analyse shared information while limiting broader exposure of the underlying data.
Microsoft is also expanding Azure Local to bring more cloud capabilities into customer-operated environments. The ability to transition workloads between connected and disconnected states is entering public preview. Additionally, DocumentDB and Blob Storage are entering public preview on Azure Local, while SQL Server on Azure Local is already generally available for connected and disconnected deployments.
On the AI front, Foundry Local is scheduled to become generally available at Microsoft Ignite for connected and disconnected operations on Azure Local. It will support local inference and agentic retrieval in natural language, backed by an expanded model catalog from Microsoft, partner networks, and the open-source community.
To further protect data during processing, Confidential Virtual Machines on Azure Local is now generally available, leveraging hardware-encrypted memory and isolation alongside workload attestation before sensitive keys or data are released.
While these additions give enterprises greater flexibility, industry analysts emphasize that localized deployments do not automatically erase vendor dependency.
“Depending upon the deployment model, in the public cloud, enterprises get better visibility, stronger access controls, and more control over encryption keys,” said Ashish Banerjee, senior principal analyst at Gartner. “In the Sovereign Private Cloud, they can operate infrastructure locally and even disconnect selected workloads from Azure. But I would not call this complete control. Enterprises still need to check whether Microsoft remains involved in identity, software updates, management, or technical support for the services they use.”
For customers in the EU and the European Free Trade Association (EFTA), Microsoft is rolling out Data Guardian. Under this model, a trusted regional partner will provide independent oversight of remote access requests by Microsoft engineers. Simultaneously, a Production Touch Ledger will create tamper-evident logs of all technical actions performed.
Microsoft is also broadening its Sovereign Partner Ecosystem, highlighting regional operators like Bleu Cloud in France and Delos Cloud in Germany, alongside independent software vendors specializing in security, identity, observability, and backup.
Despite these additions, analysts note that technical controls cannot eliminate geopolitical dependence entirely. US hyperscalers (Amazon, Google, and Microsoft) continue to account for roughly 70% of the European cloud market, whereas European providers’ market share has contracted over the last decade.
The new measures introduced under Microsoft Sovereign Cloud do reduce risks but cannot eliminate them. “Local infrastructure, independent encryption keys, and disconnected operations make it harder for external parties to access data or disrupt services. However, Microsoft remains subject to applicable US laws, sanctions, and export restrictions. Enterprises gain greater operational independence but not complete legal or technological independence,” added Jain.
Furthermore, while sovereign cloud capabilities can offer greater control over data, infrastructure, and operations, this often comes with trade-offs in cost, performance, scalability, and access to services.
Banerjee noted that sovereignty should be assessed workload by workload, rather than applied as a label to the entire cloud environment. Greater sovereignty generally comes with greater cost and responsibility. Local or disconnected environments may not receive every Azure service at the same time. Enterprises may also need dedicated infrastructure, specialist skills and separate continuity arrangements. On top of that, they may lose some elasticity and economies of scale.
“So, I would not recommend maximum sovereignty for every workload. Reserve it for workloads where the legal, operational or geopolitical risk genuinely justifies the additional complexity,” added Banerjee.
Few CIOs are likely to suddenly decide to give an AI agent broad authority over corporate systems. The more likely path to autonomy is far less dramatic. An agent starts with a narrow task. It proves useful, so another system is connected. It then needs more data, so another permission is added. Then human approval becomes a bottleneck, so the organization removes it, and exceptions that once went to people start being handled automatically.
Each decision may be sensible in isolation. Taken together, however, they can leave an agent with considerably more authority than anyone originally envisioned.
That’s the danger of autonomy creep. And as enterprises move from AI systems that recommend actions to agents that execute them, CIOs need to govern not only what those agents were originally designed to do, but what they have gradually become capable of doing.
Jim Taylor, president and chief product and strategy officer at identity security company RSA, sees a familiar pattern. Organizations typically create an agent for a specific purpose and give it the permissions necessary to accomplish the task. Later, the job expands.
“They just give it tasks, permissions; it just sits there and accumulates things,” he says. “Over time, it develops potentially toxic combinations. It gets too much access and can see too much data. It can do too many things.”
Jim Taylor, RSA
RSA
Humans accumulate permissions, too, of course, but organizations have developed processes around employees joining, changing roles, and eventually leaving. Agents may not have the same natural lifecycle. “They never give permissions back,” Taylor says.
Nancy Wang, CTO of 1Password, argues that CIOs should treat each expansion in an agent’s responsibilities as something more consequential than another feature. “Every time you expand the scope of the agent’s effective authority, you’re essentially adding a new authorization boundary,” she says.
That distinction between capability and authority is important. An agent may be technically capable of accessing a system, changing data, or executing a transaction, but that doesn’t mean the enterprise should automatically authorize it to do so.
Ironically, autonomy creep often occurs when AI works well. Paul Dongha, head of responsible AI and AI strategy at NatWest Group, sees that potential at the bank.
NatWest deployed agents in complaint handling, he says. Parts of the workflow that involve retrieving information from systems can be automated relatively easily, while humans continue to make the final decision on whether to uphold or reject a complaint.
If a system consistently produces the correct result, human reviewers may begin treating approval as a formality. The organization may eventually decide that the check is no longer necessary. “The temptation is to give over to the AI because it’s so reliable,” Dongha says.
NatWest sometimes reduces checking when an AI system demonstrates sufficiently high performance, he adds, but the decision is deliberate rather than incremental. It involves stakeholders and risk managers, and the system continues to be monitored afterward.
That’s a critical distinction for CIOs. Removing an approval step doesn’t necessarily represent poor governance. Removing it because everybody has gradually stopped paying attention to it does.
The warning sign is when a human moves from making a meaningful decision to merely rubber-stamping one — and then entirely disappears from the workflow without anyone reassessing the authority the agent has gained.
As agents acquire more authority, CIOs can’t rely solely on the agent to respect instructions about what it should and shouldn’t do. At 1Password, Wang advocates what she calls verified loops. Before an agent begins work, humans define its task, which tools it may use, what evidence will demonstrate that it completed the task successfully, and which actions it should never perform.
A separate verifier then compares evidence from the agent’s actions against that human-defined policy. “What you want to avoid here is something recursive where you have a fox guarding a henhouse, or the agent looking over its own work,” Wang says.
Nancy Wang, 1Password
1Password
The principle applies beyond software development. A model might be instructed never to transfer more than a certain amount of money, delete a production environment, or access particular data. But if violating that rule would have serious consequences, the limit should also exist outside the model.
RSA’s Taylor makes the same distinction from the identity and access side. “Model-level guardrails are necessary, but not sufficient,” he says. “The real limits need to sit outside of that in enterprise control.” He argues for layered controls around agents, including identity policies, provisioning rules, lifecycle policies, governance, monitoring, and attestations.
In other words, CIOs shouldn’t ask an AI agent to exercise restraint where the infrastructure could simply prevent the unauthorized action.
Traditional access governance often operates on a schedule. A manager might review an employee’s permissions quarterly and determine whether those permissions are still appropriate. Agents may evolve too quickly for that approach. “Agentic identities are dynamic,” Taylor says. “So we need a mechanism that certifies them continuously, rather than performing periodic reviews.”
If an agent suddenly gains access to another database, for example, that event itself could trigger reassessment. Organizations should determine whether the new permission is appropriate or creates a dangerous combination with access the agent already possesses. “Privilege and authority should never be permanent,” Taylor adds. “They should always be reviewed.”
Wang favors an even more granular approach of task-based authorization. Rather than giving agents standing access or long-lived credentials, organizations can make authorization decisions at runtime and provide short-lived credentials only when required for a particular task. “We espouse no standing privileges,” she says. “No long-lived credentials.”
That effectively applies least privilege at agent speed. A change in task becomes an opportunity to reassess whether the agent still needs the access it’s requesting.
Technical controls, however, don’t eliminate the organizational question of accountability. Taylor believes enterprises should be able to associate agents with a responsible identity. If an employee creates an agent, the organization needs to know who’s responsible for it, what its purpose is, and what happens to it if that person changes roles or leaves. “Every agent action needs to be attributed to a responsible identity,” he says.
Ownership also needs a lifecycle. An agent that remains important after its creator leaves should be transferred to another designated owner rather than simply continuing indefinitely.
Dongha makes another distinction. For AI embedded in a business process, he believes the business owner should remain accountable because that person owns the customer relationship, the process, and often the P&L. “It’s the business owner who has the responsibility because their people run it, and they’re responsible for the customer relationship,” he says.
That creates a useful division of roles for CIOs: IT may provide the infrastructure and controls, but the business can’t outsource accountability for the consequences of autonomous decisions.
The challenge becomes more complicated when agents begin working together. An individual agent might remain entirely within its permitted scope while contributing to an overall outcome nobody intended. One agent generates an output. Another accepts it as input and takes an action. A third agent then acts on the result. Again, the individual actions may all appear legitimate.
Dongha says enterprises therefore need to test not only individual agents but the entire network as a system. “You need something overarching on top of a multi-agent system,” he says.
Paul Dongha, NatWest Group
NatWest Group
NatWest is working on an AI control tower that would give authorized users visibility into a network of agents, including messages passed among them and the guardrails surrounding those interactions. Dongha considers governance of multi-agent systems an emerging problem that nobody has fully solved yet.
Wang also points to a related challenge that identity becomes more complicated when agents are ephemeral. An organization may spin up a temporary team of agents, tear it down, and recreate it later. If something goes wrong, the enterprise needs to reconstruct which agent made which decision during which session, and how that action propagated through the system.
That makes traceability increasingly important as autonomous systems become less like individual applications and more like networks of interdependent actors.
Before an agent goes into production, Taylor says CIOs should be able to answer several basic questions: Who owns it, what can it do, what’s its maximum authority, how will the organization know if that scope changes, and which actions require a human decision.
Other industries don’t need to replicate the governance machinery of a large bank, but they can borrow one important principle: controls should be built into the system from the beginning. “Governance shouldn’t be a check at the end,” Dongha says. “It needs to be embedded when you build an AI system.”
For CIOs, that may be the central lesson of autonomy creep. The question isn’t simply whether an agent can be trusted today but whether the organization has mechanisms to notice when tomorrow’s agent has quietly acquired authority that nobody ever intended to give it.
DHS has ordered DeepSeek removed from federal civilian IT systems over security concerns. Learn what the restrictions mean for agencies and technology vendors.
The post DHS Orders DeepSeek Removed from Federal IT Systems Over Security Risks appeared first on TechRepublic.
The emerging challenge with AI agents isn’t simply preventing attacks from external rogue AI agents; it’s accounting for what your own agents are doing on the inside. AI agents are already rewiring how organizations operate, and we’re only in the first inning. Recent incidents, however, have exposed the risk for individuals and enterprises alike. This is happening because organizations are moving fast and pushing themselves to move even faster.
In this environment, AI agents have created edge cases that reveal real risks when deployed without sufficient security controls. And further, no one can predict the impact of advances coming over the next six months. All of this reinforces the need to get security fundamentals right. If you’re building with AI agents right now, you know the real challenge isn’t just getting them to run. It’s knowing what happens when they act on their own. That starts with three questions. What agents do I have in my environment? What are they doing? And what are they capable of doing?
Agentic governance is problematic because enterprises must track a sprawling agent landscape AND ensure each agent is built and deployed securely. Also, agentic adoption is frictionless. Creating an agent is only one API call away from a model provider. Vibe coding means any curious employee can build an agent. This triple proliferation of vendor agents, platform native agents and homegrown agents compounds the problem.
There is the famous security adage that you can’t protect what you can’t see. Right now, most organizations lack basic visibility into what agents are in their environments. According to Anthropic, 82 percent of organizations have discovered unregistered agents in their network, and 57 percent already deploy agents for multi-stage workflows.
In addition to basic visibility, enterprises also need to understand other related attributes, such as who onboarded the agents, what access privileges they have and how they are behaving as designed. For example, an over-provisioned agent, like an email agent with 50 tools attached, would be suspect and worth investigating.
Security teams also need one view of their agents across every platform they run on, because each platform has its own permissions and its own telemetry. When each agent has a single record, security teams can connect what it can do on one platform with what it can do on another, and compare both with what the agent was built to do. That’s how seemingly routine activities add up to an attack chain. An agent might read customer records in one cloud, hold a credential for a payments API in another and be able to send messages from a SaaS application. No single platform sees a problem, but together they form a path to move sensitive data out of the company that the agent’s intended scope never required. Similarly, an agent that gains permissions across several platforms over time might never trigger an alert in any one of them, yet the combined record shows how its reach has far exceeded its purpose.
Shift-left security helps find model or tool vulnerabilities before deployment, to find risks before an agent goes into production. That’s a necessary first step, but insufficient. Agents create tightly coupled runtime behavior-to-vulnerability. Unlike traditional software, AI agents exhibit behaviors at runtime that can’t necessarily be predicted by inspecting the code in the development phase. Code analysis can’t uncover prompt hijacking, poisoned context or goal drift.
The model has evolved from “secure the code, then ship it” to continuous behavioral oversight at execution time, layered on top of build-time governance. Pre-deployment security analysis is still required because spec-driven development enforces checkpoints before code is generated. But runtime monitoring is also necessary to detect exposures in the production environment and enforce what agents do versus what they’re scoped to do.
In practice, this means applying two layers of control to every agent. The first layer is runtime behavioral boundaries. These define what an agent can do. They determine what the agent can reach, what it can return and where it must stop and ask for human approval. This is the enforcement mechanism that operationalizes runtime governance. The second layer is identity controls. These define what an agent is allowed to touch.
Think of every agent like a scoped, temporary employee with an expiration date. You enforce this by applying familiar IAM principles to agents. Least privilege. Short-lived credentials. Multi-factor authentication for long-lived credentials. Just like an individual contributor engineer shouldn’t get access to the broader team’s payroll, a code generation agent shouldn’t be provisioned to push directly to production environments. An agent designed to answer customer queries shouldn’t get read/write access to the customer database. An IT helpdesk agent doesn’t need domain admin privileges to alter security policies.
AI-fueled exploits are breaking down CVE-based severity scoring. The status quo is security teams patching the most critical vulnerabilities first. But attackers don’t always work that way. They can now use agents to chain low-severity vulnerabilities together in ways that render individual scores meaningless while building attack paths that combine for a devastating impact.
For example, an agent might combine an information disclosure flaw, a misconfigured service account and an unpatched privilege escalation into a single attack path that reaches sensitive data, even though none of those vulnerabilities would individually trigger an urgent patch cycle. If that path leads to customer PII or financial records, organizations face regulatory action under frameworks like GDPR, HIPAA or PCI DSS. If it exposes API keys, service credentials or internal network configurations, the attacker gains a foothold for lateral movement across the environment.
Context is critical. A vulnerability in network-isolated code is much less dangerous than the same vulnerability in an internet-facing service. Rather than treating every vulnerability in isolation, the better approach is to prioritize based on exploitability within the actual runtime environment. Factor in what’s reachable, what damage an attacker could do and the business impact.
The emerging challenge with AI agents is accounting for what your own agents are doing inside your enterprise. In this environment, enterprises need visibility, runtime governance and exploitability-based prioritization. To that end, the security fundamentals you already know still matter. You should already be applying least privilege, short-lived credentials, MFA for the necessary long-lived credentials, automated pipelines and proper testing to secure your corporate environment. These controls limit what an agent can access and what damage it can do if compromised. You will also want to add the agent-specific fundamentals mentioned earlier: visibility, continuous behavioral oversight, identity controls and exploitability-based prioritization. These capabilities tell you what your agents are doing, whether they should be taking those actions and how to intervene when they drift. Together, pre-agentic security fundamentals and agent-specific best practices provide the baseline governance guardrails to secure AI agents.
Organizations that get this right can accelerate agentic deployments at scale, unlocking innovation while minimizing unintended consequences.
The White House AI task force has begun a 120-day review of cybersecurity, biosecurity and oversight, with potential implications for businesses and developers.
The post White House Begins 120-Day AI Safety Review Amid Growing Security Concerns appeared first on TechRepublic.
AI agents expose gaps in enterprise access controls and audit trails. Here's how security leaders can rethink identity, data governance, and oversight.
The post Your Security Stack Knows Who Logged In. But Does It Know What AI Agents Do Next? appeared first on TechRepublic.
Eight US states have sued OpenAI over alleged ChatGPT safety risks to children. Here's what the lawsuits claim and how the company is responding.
The post Eight US States Sue OpenAI Over Alleged ChatGPT Safety Risks to Children appeared first on TechRepublic.
Companies routinely rely on authenticating the identities of humans who access sensitive company data across their networks. There’s an established and accepted set of protocols, done upfront and in a zero-trust approach – either we know who you are and let you in, or you’re locked out. The same framework needs to be applied to AI agents, posthaste.
Companies are increasingly using AI agents developed in-house, or by their vendors, partners, or clients, or, unfortunately, criminals. This can quickly add up to thousands of agents across their infrastructure and supply chains. Non-human identities outnumber human ones 45 to 1 on average in the enterprise and 144 to 1 in cloud-native environments, an increase of 44% in one year, per a recent report.
The scale and speed of this growth are beyond the ability to manually track these agents. And they have limited ability to authenticate these agents’ identities. The risks are growing. Without proper authentication, untrusted agents could access sensitive financial or customer data and share it, purposely or not, with an outside vendor, and thereby expose both the customer and the company to serious legal and compliance risk.
Many companies and security professionals are focusing on what AI agents are allowed to do in terms of permissions. But before doing that, there’s a more fundamental question they need to ask each agent: Whose agent is this? And do I trust the owner/creator? About 68% of security professionals can’t tell legitimate AI agents from malicious actors. And that question isn’t addressed at the growing scale that AI is reaching.
In email, we have seen a similar story. Emails faced challenges identifying whether a sender was really who they said they were. Due to how it originated, email had no built-in authentication framework. This led to 89% of all phishing attacks having one common denominator: the sender was lying about who they were. To address this, in 2015 my cofounder and I launched a company, Valimail (valid email), to automate the emerging email authentication standard, DMARC. If I received an email claiming to be from Bank of America, DMARC checks the sender’s DNS records to authenticate whether BofA actually authorized it. Today, 100,000s of companies use the DMARC authentication standard for their inbound and outbound email.
Fast forward to today, and these ideas and patents are directly applicable to the world of AI. DMARC doesn’t rely on spotting AI-generated deception. Instead, it uses DNS as the source of truth, which is much harder to falsify.
With AI, we can take an analogous authentication approach. When you travel somewhere, you have a passport and a visa. The passport verifies who you are, and the visa says where you’re allowed to go/what you’re allowed to do. Similarly, today, an agent passport based on Domain Zero trust authentication can identify who the agent is (its provenance) and authorize it for the system that the agent is trying to access. Then systems can verify what an agent is allowed to do (equivalent to a visa). But first, when an agent shows up claiming to represent a vendor, the receiving system verifies its identity by checking that vendor’s DNS, applying a strict zero-trust yes/no authentication – zero ambiguity, and importantly, creating an audit trail to hold the creator accountable for any actions the Agent takes.
Given the “bad” agents outnumber “good” agents 10-100:1, a zero-trust approach focuses on the limited good/approved and the rest (near infinite bad) are rejected. Stakes are High
Without this kind of authentication, many scenarios can unfold. Almost half of employees who are using unapproved AI agents are sharing HR-related information with them and 39% transmit confidential company documents. But only 34% of organizations use the same security controls for agents as they do for human employees.
Consider this: An agent claiming to represent a third-party vendor, such as a marketing automation platform, gets access because no one confirmed it was indeed that approved vendor’s agent making the request. It accesses financial records or customer data, either downloads or modifies records, and by the time this is noticed, the damage is done.
This is a quickly moving risk. Fake invoices can appear to be authentic, and they can be generated at a scale no human fraud operation could match. Authentication used to take a phone call or a manual check. But this now has to happen instantly and automatically, at scale.
Some may think email authentication is an old problem. DMARC, they say, is a decade-old fix, and the real conversation today is about AI. About 94% of organizations are dealing with increased machine identities, largely due to AI agent deployment, per Gartner, and machine identities now outnumber human accounts.
But the most common form of phishing that occurs today is when AI convincingly pretends to be someone who the target already knows and trusts. The tools for generating that deceptive attack have vastly improved. Yet the proven process for authenticating identity hasn’t kept pace, at least not for agents. This is why identity authentication is no longer optional and is now a precondition for everything else that companies want to safely do with AI agents.
Identity alone doesn’t solve the whole security problem. In addition, once an agent is authenticated, there are additional systems to verify what the agent is allowed to do and whether the content it produces can be trusted. Those questions come after this one. But both rely on the first one: Who is this?
To build a reputation, you need a known identity. Once a system knows which vendor an agent represents, it can start tracking the vendor’s behavior over time and someone becomes accountable for what its agents do. And decide if the vendor should be approved and for which actions (visas). But without strong, repeatable identity authentication of agents, other security and compliance measures for agents can’t function. It’s this foundation that supports the rest of the security stack. You can’t hold an agent accountable or decide what it should be able to do until you know who it actually is. Identity is the foundation on which the rest of AI agent security must be built.
The FTC has subpoenaed OpenAI and Anthropic over potential consumer protection violations. An AI security expert explains the risks of enterprise AI agents.
The post FTC Demands Answers From OpenAI and Anthropic About AI Safety appeared first on TechRepublic.
The EU is questioning OpenAI and Anthropic over AI agent security incidents, raising questions about the AI Act's reach into model development and testing.
The post EU Questions OpenAI, Anthropic Over AI Agent Security Incidents appeared first on TechRepublic.
For the past half century or more, the teams running enterprise software stacks have nurtured a vast ecosystem of tools to keep software humming along. They’ve built CI/CD pipelines, elaborate test suites, code repositories, and tools galore, all with the goal of keeping the systems up and running.
Now AI is throwing a wrench into the whole endeavor. Software is now more complex and sometimes mysteriously unpredictable. Expectations are sky high, but no one is certain what to expect in good conscience.
Luckily, dozens of companies are rushing to build new tools designed to keep agentic stacks on the straight and narrow, delivering value to everyone. The field is rapidly evolving, probably having changed since you started reading this article.
Some of the tools involved in the emerging agent stack aren’t that different from what enterprise teams already have in store. In the end, LLMs are just piles of bits that need to be loaded into a processing chip, just like old-fashioned software. Many of the old tools that watch for overloaded nodes or overpacked disk drives are just as valuable as ever, perhaps more so because LLMs chew through storage even faster. And some of the companies tackling these problems are the same steadfast suppliers who have earned the trust of your DevOps teams; they’ve just extended their own offerings to handle LLMs and AI.
Other challenges, however, are quite different. For example, LLMs are terrifyingly non-deterministic, a real challenge for belts-and-suspenders engineers who traditionally tend the software. Tool vendors are building entirely new systems to handle these somewhat random deviations so that traditional DevOps teams — and their C-suites — don’t have heart attacks.
The field is also rapidly expanding in wildly divergent ways. Some vendors are building tools for developers who are creating new LLMs. Others are concentrating on the challenges that come from worrying about an LLM, day in and day out, in large scale production.
We’ve compiled lists introducing the most intriguing companies developing tools for this rapidly altering space, including tools aimed at AI orchestration; AI observability, aka AgentOps; AI evaluation and benchmarking; and AI trust, guardrail, and governance. We’ve even thrown in some vibe-coding tools for citizen development and AI workflow tools for business processes.
While it was relatively easy to split the field into distinct lists, that process will only grow more challenging because the tools — and categories — are starting to overlap. As vendors evolve their strategies, this overlap will grow further, with a variety of entries tackling a wider range of challenges, vying for a greater percentage of CIOs’ AI middleware and operational budgets.
To help you make sense of this complex tool landscape, here are the most common categories vendors are using to identify what their products do, each of which maps to what are becoming the key components IT needs to keep their AI and agentic stacks productive and humming.
Vendors often use the term “observability” to refer to the most basic of data collection work — that is, testing to make sure the various machines are running, the RAM isn’t too full, latency is acceptable, and the logs aren’t filled with error codes.
Watching these metrics isn’t much different from caring for an old-school collection of microservices, and often the tools aren’t much different, either. Many of the most popular tools are the same dominant market leaders that have been extended to collect data from the various GPUs, too.
Leading vendors: DataDog, Dynatrace, New Relic, Splunk
Upstarts: Ariz, LangSmith, Braintrust, Galileo
Open-source alternatives: OpenTelemetry, Phoenix, Langfuse
Many AI pipelines are often more complex than regular microservice architectures, and that means they need someone to carefully watch the bill. Plus, as attention to AI ROI rises, capturing spend is essential — especially given how quickly AI tokens can add up.
Cost engineering tools can integrate information from various portals like OpenRouter and compare it against internal hardware. Sometimes breaking down these comparisons is difficult because the right way to allocate costs to each user isn’t as obvious, even though they’re often just one number. These accounting operations tools will become even more important as LLM bills grow.
Leading vendors: CloudZero, Finout
Upstarts: Amnic, Bifrost, Portkey
Open-source alternatives: OpenCost, Langfuse
When metrics and telemetry begin to focus on the LLM, the word “AgentOps” starts to appear. It’s just like DevOps but it’s concerned with watching token counts, tracking the performance of RAG databases, and collecting traces for debugging purposes.
When the work is truly agentic, AgentOps must also juggle the state that agents generate while looping through their various activities. Agents aren’t just a one-and-done call to an LLM. They’re software packages that need to store information as they work toward a final answer.
This job of an AgentsOps team gets trickier because these areas are where the models often behave non-deterministically. Is some model hallucinating? Are an agent’s prompt chains running to completion? The tools in this space track all these metrics, often in a dashboard with nice graphs that make problems stand out.
Leading vendors: Databricks, Datadog, Dynatrace
Upstarts: Arize, Braintrust, Comet
Open-source alternatives: Langfus, LlamaIndex
The buzzword “evaluation” starts to appear alongside tools that are used in the earlier stages of development, when various models and prompt combinations need to be tested again and again.
In many cases, these tools use another model to evaluate the quality of the answer from the first model, as well as to keep an eye out for potential hallucinations or other failures. In some cases, the tools query humans who sit in the evaluation loop to give their opinions of how well the model is performing. Good tools can watch the agents continuously and reroute any sketchy behavior to the human-in-the-loop.
Many of the vendors use the word “evaluation” to describe measuring just how well a model is performing for a particular business application. That is, how well it’s working for what your boss wants it to do.
Leading vendors: Databricks, SuperAnnotate, Scale AI
Upstarts: Braintrust, Galileo AI, MaximAI
Open-source alternatives: Langfuse, Prompfoo, Tagas
The word “benchmarking” is similar to “evaluation” in AI vendor literature, but it is usually found more in the tools used by the creators of the models. There are a number of public benchmarks filled with standard queries, and the model developers rank themselves on how well they do solving these problems. This often generates a raw score that is then compared to another’s raw number. Some teams create their own benchmarking problem sets and use them during evaluation to assess how well models might meet their needs.
Leading vendors: Hugging Face, Databricks
Upstarts: Reka AI, Patronus AI
Open-source alternatives: Promptfoo, DeepEval
The tools that wear the “trust” benchmark are often meant to help evaluate how well a model is protecting the needs of any business that is deploying it. That is, can they trust it to deliver accurate and useful answers without hallucinations, data leakage, inappropriate behavior, or worse? Prompts with malicious instructions are especially tricky to detect. Some enterprises want a trust score that measures the consistency of the answers. Overall, tools that target the “trust” metric are often utilized more during development and early deployment.
Leading vendors: Azure AI Content Safety, Bedrock Guardrails, Vertex AI Search
Upstarts: Patronus AI, Arthur AI
Open-source alternatives: NeMo Guardrails, Llama Guard
The word “guardrails” is usually applied to tools that are part of the deployment stack. That is, they sit as watchdogs and make sure the LLM doesn’t spin out of control. They also ensure the model is trustworthy, but they’re more tuned to operating in real-time than those aimed at trust, as outlined above.
These tools are often filled with fixed rules, such as scanning for social security numbers or watching for noxious words. The challenges get trickier because agents are executing software, and that means the guardrails must do more than analyze outputs. They also need to manage the execution environment, often running in a sandbox, and watch for exploits and other malicious behavior. They need to validate and double-check not just code, but the OS-level commands for manipulating files and processes.
Leading vendors: Azure AI Content Safety, Bedrock Guardrails, Vertex AI Search
Upstarts: Patronus AI, Arthur AI
Open-source alternatives: NeMo Guardrails, Llama Guard
Many systems rely on a vector database to store embedded data to chunks of trustworthy documents. These RAG databases need much of the same care as regular databases, and the tools for their care are often similar. PostgreSQL, for example, has a vector addition that needs the same kind of scrutiny from a database administrator as a regular PostgreSQL database.
Leading vendors: MongoDB, Elasticsearch
Upstarts: Pinecone, Weaviate
Open-source alternatives: Milvus, PostgreSQL
Increasingly, AI systems employ models in constellations, and a gateway router is what examines a prompt to determine which model or agent is best for answering the query. Moreover, as agents undertake their work, the router can distribute the agent’s various subcalls to the fastest, cheapest, or most accurate LLM or subagent.
Watching over this stage in the pipeline requires a different set of intelligence because the questions can often involve both applicability and cost. Sometimes this layer must balance the extra cost versus the need for a better answer.
Leading vendors: Cloudflare, AWS, Google Cloud, Azure
Upstarts: OpenRouter, ModelsLab, Respan
Open-source alternatives: LangChain, Bifrost, LiteLLM
Who gets to use the model? Who is paying? Many access control tools are the same ones that were part of standard APIs or web services. Is there an API key? Is the account below its budget? The tools watch for overspending because both the customer and the boss want to make sure the money keeps flowing and no tokens are wasted.
Leading vendors: Auth0, Apigee
Upstarts: Amnic, Portfoundry
Open-source alternatives: LiteLLM, Bifrost
Some industries have explicit compliance requirements dictated by law. Some have their own internal controls. Tools devoted to compliance are often a subset of the ones devoted to guardrails. But they’re often treated separately from guardrail tools because they’re more likely to be of interest to lawyers and that makes the stakes higher.
Leading vendors: ServiceNow, IBM
Upstarts: VerifyWise, Norm AI
Open-source alternatives: Aether, Giskard
Fresh off its $1.1 billion acquisition of AI startup Sana, Workday is making another bet on enterprise AI, this time targeting the governance challenge. The company said it will acquire agentic identity platform C1.ai to help customers manage and secure AI agents operating across enterprise applications.
The acquisition will build on Workday’s Agent System of Record, which gives enterprises a centralized view of their AI agents, by adding controls over what those agents can access across Workday and other enterprise systems, the company said.
That would allow enterprises to grant agents the permissions they need to perform specific tasks and revoke them when those tasks are completed, rather than risk agents retaining access to data and applications they no longer need, it added.
The combined capabilities could also help streamline access management for employees as they join an organization, change roles, or leave, reducing the manual work involved in managing permissions while limiting potential security gaps, it further said.
These governance and identity management capabilities become increasingly important as Workday, alongside the acquisition, also continues to expand the role of AI agents across its suite.
During the week, the company announced several updates to Sana, including making it the new home interface for its applications.
The interface brings tasks, decisions, meetings, and agent activity into one place, allowing employees to use natural-language prompts to get Workday agents to complete tasks rather than simply retrieve information, the company said.
The company also extended Sana into third-party tools, including Microsoft Teams, Slack, and Microsoft Copilot, allowing employees to invoke Workday capabilities from the applications they already use.
That expanded ability to invoke Workday agents across applications underscores the need for enterprises to govern what these agents can access and do across systems. C1.ai’s acquisition is expected to close by the end of January.
Entrar