For such a rapid innovation cycle, AI has been given unprecedented levels of responsibility. According to the Stanford AI Index 2026, 88% of organizations used AI in 2025, with 70% using generative AI in at least one business function. Most analysts agree that this technology, particularly when it comes to generative and agentic AI, is yet to reach full maturity, yet it’s already making itself indispensable to most enterprises. Employees expect LLM-based copilots to respond as readily as any other business application, customers are increasingly exposed to AI as part of their user experience, and emerging agentic models need to communicate continuously with applications and infrastructure as they crunch data and carry out tasks. Any hint of delay within those interactions has the potential to disrupt productivity, sow mistrust in the technology, and limit any return on investment (ROI).

Most businesses now inhabit a multi-cloud environment that spans countries and continents. When an AI request depends on information stored in one cloud environment, processing capacity hosted in another, and an application delivered somewhere else entirely, latency becomes the deciding factor. Each network hop, particularly through public Internet pathways, increases response time. Repeat these delays across hundreds, thousands, or even millions of individual requests – from both humans and AI agents – and the whole organization becomes artificially hampered. Sometimes connectivity becomes so hampered that IT teams must step in and spend valuable time rectifying it.

A recent Censuswide survey commissioned by DE-CIX found that IT teams in the US and UK spend an average of 11.5 hours every week resolving cloud connectivity issues – that’s more than a full working day each week spent on network troubleshooting. This is the “hidden productivity tax” that organizations are now paying, and it may be why the ROI for AI initiatives feels lukewarm at best, and impossible to measure at worst. McKinsey’s 2025 global AI survey found that only 39% of respondents could attribute any enterprise-level EBIT impact to AI, and most of those reported a contribution below 5%. That’s why even the best models can only get you so far – compute power may command the budget, but it’s the network that determines value.

A tax on productivity

Consider what happens when an employee asks an AI assistant to carry out a simple task like summarizing an internal document or doing some number crunching. The application may need to verify the employee’s identity, locate the relevant information, send it to a model operating in another cloud environment, and return the response through the user-facing application. An agent completing a more involved task could make this journey repeatedly as it gathers information and interacts with other systems. If one connection is slow or unreliable, the delay follows the request all the way back to the user, and if that connection fails, a whole team may be drafted in to isolate a problem that appeared inside the application but originated somewhere along the network path.

Engineers who could be improving AI services or supporting new deployments are instead tracing routes, investigating intermittent failures, and working around infrastructure that behaves unpredictably. What makes this particularly striking is the contradiction that was revealed as part of DE-CIX’s survey. While enterprises concede that more than a full working day per week is spent on network troubleshooting, as noted above, nearly all (96%) say their networks are “ready” to support future cloud and AI projects. This doesn’t prove that enterprise networks are fundamentally unprepared, but it does reveal a disconnect between strategic confidence and everyday operational experience. When more than a working day can disappear into connectivity troubleshooting each week, some of the expected productivity benefit from AI is already being spent before the technology has had an opportunity to deliver it.

AI performance and network performance are inseparable

Users don’t experience a model and its supporting infrastructure as individual sets of requests – they just experience “fast” or “slow” workloads. A cutting-edge AI model will still feel ineffective if the information it needs travels along congested or unpredictable public transit routes, so any real-time interaction with AI will become an exercise in frustration with small delays that compound over time, slowing teams down.  As AI services become more distributed, network performance is becoming inseparable from application performance, but it’s rarely given the same attention as model accuracy, compute capacity, or the cost of running AI inference.

IT leaders are beginning to take note, however, despite being a little overconfident about their network readiness. Alongside latency between cloud environments, security vulnerabilities including DDoS attacks, and downtime or reliability concerns are triggering a change in approach towards network connectivity. That’s why many are now turning to private, direct connectivity to cloud providers through an Internet or Cloud Exchange or via other means of directly connecting to the cloud providers’ private connectivity solutions, bypassing the pitfalls of the public Internet and IP transit.

Direct connectivity offers a greater degree of sovereignty, visibility, and control over data pathways that can reduce latency and enhance security. Public Internet connectivity will remain essential, of course, but business-critical AI traffic should not follow routes that can change unexpectedly or introduce additional network hops. It isn’t a universal cure for poor AI performance, but it can remove some of the uncertainty from an operating model that increasingly depends on information moving quickly and reliably between distributed environments.

Turning network performance into a business metric

The next 12 months could be a turning point for businesses looking to prove AI’s worth. CIOs simply need to account for the full cost of delivering an AI service, including the hours spent diagnosing connectivity problems and the value lost when applications respond inconsistently. In other words, they need to bring network teams into AI planning from the outset, considering their network infrastructure with the same rigor they apply to selecting AI models and projects. Private interconnection can provide more direct and predictable routes between environments, while effective observability can reveal how traffic moves, where delays accumulate, and how reliably the entire workflow performs. Without that control and visibility, network friction will remain a “hidden tax on productivity” that could stall AI deployments and weaken the case for future investment.

As enterprises continue to move AI agents and agentic applications into production, AWS says traditional observability and monitoring tools — including its own CloudWatch service —will struggle to explain why an agent behaved the way it did.

CloudWatch uses metrics, logs, and traces to monitor applications and infrastructure across accounts, regions, and services through the AWS Management Console, but that only provides part of the picture, AWS says. Understanding an agent’s behavior requires developers and operations teams to jump between agent-specific observability and evaluation tools such as those available through Amazon Bedrock AgentCore, application performance monitoring, and infrastructure monitoring in CloudWatch.

AWS is trying to eliminate that fragmentation by evolving and expanding CloudWatch with a new off-console experience named CloudWatch Omni, bringing agent, application, and infrastructure telemetry together in an application-centric setup to help enterprises investigate and understand agent behavior in context.

That means developers and operations teams can start with the application they are investigating, rather than navigating across individual AWS resources and monitoring consoles, the hyperscaler wrote in a blog post presenting CloudWatch Omni.

The new tool automatically discovers application topology, the company said, showing how its components are connected, in turn allowing developers and operations teams to query telemetry using natural language or SQL immediately without any manual setup.

Those natural language queries are supported by an AI assistant that handles discovery and guided investigation with support from the AWS DevOps Agent, which is built-in, to  automatically correlate data and identify the root cause across agents, applications, and infrastructure, it said.

How to deploy CloudWatch Omni

The application-centric approach also changes how teams get started with Omni.

For existing CloudWatch customers, the transition to Omni does not require reconfiguring their existing telemetry as logs, metrics, and traces already sent to CloudWatch become available in Omni through a unified data store that enables correlated analysis across signal types, AWS said. Existing instrumentation, dashboards, and alarms also carry forward into the new setup.

For teams new to CloudWatch and Omni, enterprises will have to import telemetry through OpenTelemetry Protocol (OTLP) endpoints and create a “space” for an application within CloudWatch Omni, which then automatically discovers the application topology and begins surfacing its dependencies and health signals.

Omni launches with support for multiple agent development frameworks, enabling teams to bring agents built using LangGraph, CrewAI, OpenAI Agents SDK, Vercel AI SDK, and AWS Strands into the same observability setup, AWS said.

It also supports independent evaluation tools, allowing teams to bring their existing agent evaluation workflows into Omni, including Braintrust, DeepEval, and Ragas, the hyperscaler added.

Enterprises also get the option of choosing the kind of Omni interface they want to work with: Operational teams can access an off-console web experience, and developers can see agent traces in Omni via native extensions for VS Code, Kiro, and Cursor. The extensions can also be used to run and trace agents locally, with no AWS account required for local development, the company said.

Omni could improve developer productivity

That ability to access Omni directly via development environments, combined with its application-centric approach should remove the day-to-day friction developers face when investigating agent behavior, in turn improving productivity, analysts said. And there are advantages for CIOs too.

“With Omni, CIOs could gain one operating view across agents, applications, and infrastructure, in turn reducing tool fragmentation,” said Stephanie Walter, practice lead of AI stack at HyperFrame Research.

That, in turn, could speed agent deployment, said Ashish Chaturvedi, executive research leader at HFS Research.

“The blocker on enterprise agent deployment right now is rarely capability, because the agents can usually do the work. The blocker is that CIOs cannot confidently answer what happens when an agent gets it wrong, how they would know, and how fast they could find out. Without an answer to that, no responsible CIO hands an agent authority over anything that touches revenue or customers,” Chaturvedi said. “Since Omni reduces the tools and time required to investigate agent behavior, it could help CIOs move agents from pilots into production by giving them greater visibility into how agents behave when they make mistakes, how quickly those issues can be identified, and what impact they could have on business operations,” Chaturvedi said.

Lock-in and cost concerns

But that simplicity has consequences, he said: “The more of your observability runs through one vendor’s layer, the more your ability to understand your own systems depend on that vendor.”

There are also potential cost and evaluation challenges that CIOs should take note of, according to Michael Leone, principal analyst at Moor Insights and Strategy.

“Agents generate a lot of telemetry because every prompt, tool call and handoff gets traced, so ingestion bills can climb faster than teams expect,” Leone said.

“Also, agent evaluations are only as good as an enterprise’s definition of a good answer, and a lot of them haven’t written that definition down yet,” he added.

Who is likely to adopt Omni and why

Enterprises with mature application and infrastructure observability environments based on Datadog, New Relic, or Grafana may not have much to gain from switching to Omni, according to Chaturvedi.

The earliest adopters, said Walter, are likely to be existing CloudWatch and Bedrock AgentCore customers because their telemetry and instrumentation already carry forward into Omni — and, she added, enterprises running multiple agent pilots may also benefit from Omni to standardize evaluation and operations.

Availability and pricing

Omni is currently available in the US East (N. Virginia), US West (Oregon), and Europe (Ireland) AWS cloud regions. However, the company maintains that this limited regional availability does not restrict its usage.

“Even while Omni runs in these initial Regions, customers can centralize telemetry from across all their accounts and Regions into a preferred Omni Region, giving them a single, unified view of their observability data at no additional cost,” an AWS spokesperson said.

Omni users must pay for data ingestion, storage, and analytics, with telemetry from AWS services available at tiered pricing. Ingestion is charged per gigabyte, and storage per gigabyte per month.

Analytics pricing is usage based, with analytics equivalent to up to five times the amount of logs or spans ingested included at no additional cost, the spokesperson said.

There’s a separate price list for AWS DevOps Agent, which is integrated into Omni.

Existing CloudWatch customers will not be automatically moved to Omni, but can opt in to create an Omni space and configure access.

This article first appeared on InfoWorld.

At their deepest level, LLMs are still a kind of magic. Even the developers who build them find them to be, channeling Winston Churchill, “a riddle, wrapped in a mystery, inside an enigma.”

That’s why everyone working with LLMs in their enterprise stack needs a way to peer into the dark mass of weights to help make sense of these numerical beasts.

Lately there’s been an explosion of tools that can assist. Companies are building platforms that sit in an agentic AI niche market that might be called “Evaluation and Benchmarking.” This tools track the best performing LLM or agentic options, testing their fit and watching over them as they chew through tokens.

With agentic AI still an emerging technology class, the boundaries between its nascent market niches are far from set. There are other sets of tools for tracking raw performance, an area that some call “AgentOps” or “Observability.” (See “19 AgentOps tools for monitoring AI activity, issues, and costs.”) And still more tools that focus on maintaining our faith in agent answers and on building controls to keep agents from straying, a niche that’s starting to be called “Trust and Guardrails.”

Some of the vendors operating in these spaces are starting in one category and then expanding into another. Others are diving as deeply as they can into their niche. The next year — no, let’s say the next few months — are bound to be fascinating as the tools improve and the various markets evolve and intermix.

For now, here’s a list, in alphabetical order, of some of the best options for any enterprise team that needs to evaluate agents and benchmark their performance. 

Braintrust

Big projects require tools that can scale to handle the large amount of dataflows required to trace and pinpoint errors. Braintrust is built to support enterprise-size efforts to deliver meaningful answers to a large collection of users. The tool’s sales literature promises to “trace everything” in order to have the right data available when it’s time to dissect a failed response. Braintrust also delivers a helpful dashboard that aggregates all this data so large errors in latency, cost, or quality can be identified quickly. An automated set of evaluation tasks can track answers and compile useful metrics for ensuring the agent stack is answering the needs of a large set of end-users.

Pricing: A free plan comes with $10 of credits. Pro plan starts at $250 and comes with more credits and a longer retention period.

Standout feature: Loop agent tracks behavior through multiple iterations for deeper debugging power.

Best suited for: Fast-moving teams iterating on prompts and product

Confident AI

Developers who rely on DeepEval but don’t want to host the code can turn to Confident AI, a cloud-based platform for fast, simple, and seamless deployment. The system adds a sophisticated UI that includes a dashboard for tracking and archiving all tests. This collaborative environment enables teams to work swiftly together without worrying about the troubles of exchanging problematic traces or other telemetry files. This makes it easier to extend the power of tools such as DeepEval to handle the continuous tracing and testing necessary in production environments. 

Pricing: A “forever free” plan offers a taste. The pay plan starts at $200 and includes features such as better automation and simulation.

Standout feature: Automated red-teaming and on-demand pen-testing helps build more secure results.

Best suited for: Enterprise teams building on established stacks that need the convenience of a collaborative environment

DeepEval

When a model finds a home in a production environment, it’s time to add unit tests that will double and triple check its behavior so the developers can iterate and the CI/CD pipeline can catch any mistakes or regressions. DeepEval delivers a set of Pytest-native Python scripts that run either locally or as part of the deployment pipeline. The tests check simple issues as well as more complicated and ephemeral problems such as hallucinations, drift, role adherence, knowledge retention, and conversation completeness. If the LLM starts to act up or turn into a toxic rogue, these tests will flag them.

Pricing: The open-source version of Confident AI’s tool is available with an Apache 2.0 license.

Standout feature: Full complement of PyTest modules watch for problems such as hallucinations or worse.

Best suited for: Teams with the depth and ability to fully embrace open-source tooling

LangSmith (from LangChain)

As agentic approaches begin to dominate, dev teams need a deep debugging tool like LangSmith, which tracks not just inputs and outputs, but all the steps an agent takes as well as the context that evolves along the way. This enables developers to pinpoint the stage or mechanism deep in the agent where latency, quality, coherence, or other agent parameters go wrong. The tool can be integrated with Python, Go, Java, or TypeScript applications or be used from a cloud-based app that offers a sophisticated UI.

Pricing: Solo accounts start for free. Paid tier ($39 per month per seat) unlocks more tracing and better support.

Standout feature: Complex agent graphs can be tracked with automated surveillance. 

Best suited for: Teams invested in the Langfuse tool stack

Langfuse

Finding the best model means feeding the same prompt to the same model, a process that’s getting only more complicated as developers build out multilayered agents that break tasks into multiple steps. Langfuse is an open-source AI tracking tool from Clickhouse, a company that specializes in curating oracular tools like databases. Teams can work together through the Langfuse platform to juggle the various prompts, traces, and answers. The system nurtures an LLM evaluation loop so that teams can find the best combinations of models and agents to solve the problem at hand.

Pricing: Open-source versions offer starter support. Core version starts at $29 per month and includes more traces, longer retention, and better support.

Standout feature: Open Telemetry functionality offers modularity and flexibility.

Best suited for: Budget-focused teams with the ability to leverage open-source ecosystems

LiveBench

Developers who want to send a set of questions to an LLM and then evaluate the performance turn to LiveBench, an open-source tool kit that’s routinely used to benchmark many models during development. Answers are deliberately not graded by other LLMs but compared against hard-coded answers. The tool can be extended, but there’s no fancy GUI. The work is done with configuration text files that specify the ground truth for evaluating the result. When you’re done, you can even contribute your questions to the general open-source project so that others can use them to guide LLM development.

Pricing: Open source

Standout feature: Frequently updated benchmarks offer contamination-free evaluations of models.

Best suited for: Teams evaluating a wide range of models in search of the best performance for their applications

Maxim AI

As the workloads grow more complex and combine multiple steps through workflow graphs, tools such as Maxim AI become more useful. Maxim AI tracks results with an end-to-end tool for evaluating and simulating agents. Prompts and agents and the trajectory they take to an answer can be endlessly simulated prior to deployment and then observed through deployment. The framework-agnostic tool links datasets and data providers to give teams the best insight into how well an agent is delivering.

Pricing: Free model offers one workspace with three-day retention. Pro plan starts at $29 per person per month with longer retention period, more logs, and features such as simulations.

Standout feature: Full simulator can test a wide range of uses and users.

Best suited for: Teams focused on delivering conversational agents

MLflow

Much of the work of developing a useful agentic solution is a long slog through endless combinations and iterations. The MLflow open-source platform is designed to optimize this process and speed it up as much as possible. It is part of a larger tool collection that follows the entire lifecycle of a model from training to deployment. The later stages of development, for instance, rely on systems such as the Prompt Registry, a kind of version control that allows prompt engineers to work through various approaches and linguistic tropes. The goal of the entire process is to deliver the evaluation cycles necessary to deliver a model up to its set of targeted tasks.

Pricing: Free and open source for self-hosted. Cloud computing charges for hosted versions.

Standout feature: Full lifecycle tracking for following models and tracking their costs

Best suited for: Enterprise teams watching a collection of machine learning and AI-based algorithms

Onyx

One of the simplest ways to build a basic chat system that incorporates local retrieval-augmented generation (RAG) knowledge bases is to download Onyx, a front-end tool that’s available as either an MIT-licensed community edition or as a commercial product with a few more features useful to larger enterprises. The RAG layer guides search, and Onyx’s developers built an open-source framework for testing RAG performance. Onyx administrators can also track what users are asking and how well they like the final result.

Pricing: A free starter plan offers limited storage and one database. Pro plan starting at $49 per month offers many more traces, larger storage, and access to features such as saved workflows.

Standout feature: Real-time search for monitoring production environments at scale

Best suited for: Enterprise with larger challenges with substantial RAG integration

Promptfoo

LLMs can fail in a number of ways. Promptfoo iterates through various tests that simulate real user interactions to simulate the types of issues an LLM might face each day. Promptfoo also focuses on some of the biggest security problems and specializes in red teaming to detect any failure points that might be exposed by a malicious user. From toxic edge states to personally identifiable information (PII) leaks, the goal is to deliver tests that will expose potential jailbreaks and failures in the guardrails.

Pricing: “Free forever” means an open-source tool with community-based support. An enterprise version offers custom deployment options and better support.

Standout feature: Automated red-teaming and prompt scrutiny helps lock down implementations.

Best suited for: Security-focused teams that are constantly evaluating and re-evaluating their product’s security.

RAGAS

When RAG databases are a key part of the agentic stack, developers turn to RAGAS to stress test the deeper mathematical corners of the retrieval mechanism. The Python library offers standard and custom metrics for evaluating the performance of the RAG storage-and-retrieval mechanism at the level of vector mathematics. These measure behaviors such as faithfulness, relevance, and totality of recall. The philosophy begins with experiments to speed development but ends with fast integration with the deployment pipeline. Instead of just doing a “vibe check” on the RAG database, developers are using a more scientific approach to test and converge on better total performance.

Pricing: Fully open source under Apache 2.0 license

Standout feature: RAG focus helps teams relying on vector databases for knowledge curation.

Best suited for: Teams with a substantial reliance on RAG databases

Rhesis AI

Many of tools in this evolving market niche are designed for hard-core developers. Rhesis AI wants to bring other stakeholders into the development cycle so they can create tests and evaluate performance, too. That means domain experts, product managers, and even C-suite suits can track how the LLMs behave in conversations. Adversarial or confrontational engagements that devolve into the edge cases that bring headaches are easy to simulate repeatedly to optimize responses. The platform is designed to test all stages of development in a way that’s accessible to all stakeholders.

Pricing: Said to be “open source first” but with enterprise plans for those that need it.

Standout feature: The focus on putting humans in the loop is ideal for applications that require input from meat-based intelligence.

Best suited for: Applications requiring more collaboration with domain experts

Vellum

Anyone who needs a personal assistant can turn to Vellum to help build one that is trained on your data. Along the way, you will evaluate performance using its elaborate testing framework that tracks performance against any of the metrics and use cases you supply. Real-time dashboards track performance using metrics such as token usage costs, latency, or response quality. Multiple teams can work in parallel with version controls that allow iteration and competition. The end result is an agent that’s tuned to your needs.

Pricing: A basic free tier for experimentation. The Mighty starts at $30 per month and comes with more storage and compute credits.

Standout feature: End-to-end integration simplifies managing new development.

Best suited for: Cross-functional teams looking for a centralized solution with wide integration