Visualização normal

Antes de ontemCybersecurity News
  • ✇Security | CIO
  • Looking to avoid agentic failure? These 13 AI evaluation tools will help
    At their deepest level, LLMs are still a kind of magic. Even the developers who build them find them to be, channeling Winston Churchill, “a riddle, wrapped in a mystery, inside an enigma.” That’s why everyone working with LLMs in their enterprise stack needs a way to peer into the dark mass of weights to help make sense of these numerical beasts. Lately there’s been an explosion of tools that can assist. Companies are building platforms that sit in an agentic AI nich
     

Looking to avoid agentic failure? These 13 AI evaluation tools will help

20 de Agosto de 2026, 07:01

At their deepest level, LLMs are still a kind of magic. Even the developers who build them find them to be, channeling Winston Churchill, “a riddle, wrapped in a mystery, inside an enigma.”

That’s why everyone working with LLMs in their enterprise stack needs a way to peer into the dark mass of weights to help make sense of these numerical beasts.

Lately there’s been an explosion of tools that can assist. Companies are building platforms that sit in an agentic AI niche market that might be called “Evaluation and Benchmarking.” This tools track the best performing LLM or agentic options, testing their fit and watching over them as they chew through tokens.

With agentic AI still an emerging technology class, the boundaries between its nascent market niches are far from set. There are other sets of tools for tracking raw performance, an area that some call “AgentOps” or “Observability.” (See “19 AgentOps tools for monitoring AI activity, issues, and costs.”) And still more tools that focus on maintaining our faith in agent answers and on building controls to keep agents from straying, a niche that’s starting to be called “Trust and Guardrails.”

Some of the vendors operating in these spaces are starting in one category and then expanding into another. Others are diving as deeply as they can into their niche. The next year — no, let’s say the next few months — are bound to be fascinating as the tools improve and the various markets evolve and intermix.

For now, here’s a list, in alphabetical order, of some of the best options for any enterprise team that needs to evaluate agents and benchmark their performance. 

Braintrust

Big projects require tools that can scale to handle the large amount of dataflows required to trace and pinpoint errors. Braintrust is built to support enterprise-size efforts to deliver meaningful answers to a large collection of users. The tool’s sales literature promises to “trace everything” in order to have the right data available when it’s time to dissect a failed response. Braintrust also delivers a helpful dashboard that aggregates all this data so large errors in latency, cost, or quality can be identified quickly. An automated set of evaluation tasks can track answers and compile useful metrics for ensuring the agent stack is answering the needs of a large set of end-users.

Pricing: A free plan comes with $10 of credits. Pro plan starts at $250 and comes with more credits and a longer retention period.

Standout feature: Loop agent tracks behavior through multiple iterations for deeper debugging power.

Best suited for: Fast-moving teams iterating on prompts and product

Confident AI

Developers who rely on DeepEval but don’t want to host the code can turn to Confident AI, a cloud-based platform for fast, simple, and seamless deployment. The system adds a sophisticated UI that includes a dashboard for tracking and archiving all tests. This collaborative environment enables teams to work swiftly together without worrying about the troubles of exchanging problematic traces or other telemetry files. This makes it easier to extend the power of tools such as DeepEval to handle the continuous tracing and testing necessary in production environments. 

Pricing: A “forever free” plan offers a taste. The pay plan starts at $200 and includes features such as better automation and simulation.

Standout feature: Automated red-teaming and on-demand pen-testing helps build more secure results.

Best suited for: Enterprise teams building on established stacks that need the convenience of a collaborative environment

DeepEval

When a model finds a home in a production environment, it’s time to add unit tests that will double and triple check its behavior so the developers can iterate and the CI/CD pipeline can catch any mistakes or regressions. DeepEval delivers a set of Pytest-native Python scripts that run either locally or as part of the deployment pipeline. The tests check simple issues as well as more complicated and ephemeral problems such as hallucinations, drift, role adherence, knowledge retention, and conversation completeness. If the LLM starts to act up or turn into a toxic rogue, these tests will flag them.

Pricing: The open-source version of Confident AI’s tool is available with an Apache 2.0 license.

Standout feature: Full complement of PyTest modules watch for problems such as hallucinations or worse.

Best suited for: Teams with the depth and ability to fully embrace open-source tooling

LangSmith (from LangChain)

As agentic approaches begin to dominate, dev teams need a deep debugging tool like LangSmith, which tracks not just inputs and outputs, but all the steps an agent takes as well as the context that evolves along the way. This enables developers to pinpoint the stage or mechanism deep in the agent where latency, quality, coherence, or other agent parameters go wrong. The tool can be integrated with Python, Go, Java, or TypeScript applications or be used from a cloud-based app that offers a sophisticated UI.

Pricing: Solo accounts start for free. Paid tier ($39 per month per seat) unlocks more tracing and better support.

Standout feature: Complex agent graphs can be tracked with automated surveillance. 

Best suited for: Teams invested in the Langfuse tool stack

Langfuse

Finding the best model means feeding the same prompt to the same model, a process that’s getting only more complicated as developers build out multilayered agents that break tasks into multiple steps. Langfuse is an open-source AI tracking tool from Clickhouse, a company that specializes in curating oracular tools like databases. Teams can work together through the Langfuse platform to juggle the various prompts, traces, and answers. The system nurtures an LLM evaluation loop so that teams can find the best combinations of models and agents to solve the problem at hand.

Pricing: Open-source versions offer starter support. Core version starts at $29 per month and includes more traces, longer retention, and better support.

Standout feature: Open Telemetry functionality offers modularity and flexibility.

Best suited for: Budget-focused teams with the ability to leverage open-source ecosystems

LiveBench

Developers who want to send a set of questions to an LLM and then evaluate the performance turn to LiveBench, an open-source tool kit that’s routinely used to benchmark many models during development. Answers are deliberately not graded by other LLMs but compared against hard-coded answers. The tool can be extended, but there’s no fancy GUI. The work is done with configuration text files that specify the ground truth for evaluating the result. When you’re done, you can even contribute your questions to the general open-source project so that others can use them to guide LLM development.

Pricing: Open source

Standout feature: Frequently updated benchmarks offer contamination-free evaluations of models.

Best suited for: Teams evaluating a wide range of models in search of the best performance for their applications

Maxim AI

As the workloads grow more complex and combine multiple steps through workflow graphs, tools such as Maxim AI become more useful. Maxim AI tracks results with an end-to-end tool for evaluating and simulating agents. Prompts and agents and the trajectory they take to an answer can be endlessly simulated prior to deployment and then observed through deployment. The framework-agnostic tool links datasets and data providers to give teams the best insight into how well an agent is delivering.

Pricing: Free model offers one workspace with three-day retention. Pro plan starts at $29 per person per month with longer retention period, more logs, and features such as simulations.

Standout feature: Full simulator can test a wide range of uses and users.

Best suited for: Teams focused on delivering conversational agents

MLflow

Much of the work of developing a useful agentic solution is a long slog through endless combinations and iterations. The MLflow open-source platform is designed to optimize this process and speed it up as much as possible. It is part of a larger tool collection that follows the entire lifecycle of a model from training to deployment. The later stages of development, for instance, rely on systems such as the Prompt Registry, a kind of version control that allows prompt engineers to work through various approaches and linguistic tropes. The goal of the entire process is to deliver the evaluation cycles necessary to deliver a model up to its set of targeted tasks.

Pricing: Free and open source for self-hosted. Cloud computing charges for hosted versions.

Standout feature: Full lifecycle tracking for following models and tracking their costs

Best suited for: Enterprise teams watching a collection of machine learning and AI-based algorithms

Onyx

One of the simplest ways to build a basic chat system that incorporates local retrieval-augmented generation (RAG) knowledge bases is to download Onyx, a front-end tool that’s available as either an MIT-licensed community edition or as a commercial product with a few more features useful to larger enterprises. The RAG layer guides search, and Onyx’s developers built an open-source framework for testing RAG performance. Onyx administrators can also track what users are asking and how well they like the final result.

Pricing: A free starter plan offers limited storage and one database. Pro plan starting at $49 per month offers many more traces, larger storage, and access to features such as saved workflows.

Standout feature: Real-time search for monitoring production environments at scale

Best suited for: Enterprise with larger challenges with substantial RAG integration

Promptfoo

LLMs can fail in a number of ways. Promptfoo iterates through various tests that simulate real user interactions to simulate the types of issues an LLM might face each day. Promptfoo also focuses on some of the biggest security problems and specializes in red teaming to detect any failure points that might be exposed by a malicious user. From toxic edge states to personally identifiable information (PII) leaks, the goal is to deliver tests that will expose potential jailbreaks and failures in the guardrails.

Pricing: “Free forever” means an open-source tool with community-based support. An enterprise version offers custom deployment options and better support.

Standout feature: Automated red-teaming and prompt scrutiny helps lock down implementations.

Best suited for: Security-focused teams that are constantly evaluating and re-evaluating their product’s security.

RAGAS

When RAG databases are a key part of the agentic stack, developers turn to RAGAS to stress test the deeper mathematical corners of the retrieval mechanism. The Python library offers standard and custom metrics for evaluating the performance of the RAG storage-and-retrieval mechanism at the level of vector mathematics. These measure behaviors such as faithfulness, relevance, and totality of recall. The philosophy begins with experiments to speed development but ends with fast integration with the deployment pipeline. Instead of just doing a “vibe check” on the RAG database, developers are using a more scientific approach to test and converge on better total performance.

Pricing: Fully open source under Apache 2.0 license

Standout feature: RAG focus helps teams relying on vector databases for knowledge curation.

Best suited for: Teams with a substantial reliance on RAG databases

Rhesis AI

Many of tools in this evolving market niche are designed for hard-core developers. Rhesis AI wants to bring other stakeholders into the development cycle so they can create tests and evaluate performance, too. That means domain experts, product managers, and even C-suite suits can track how the LLMs behave in conversations. Adversarial or confrontational engagements that devolve into the edge cases that bring headaches are easy to simulate repeatedly to optimize responses. The platform is designed to test all stages of development in a way that’s accessible to all stakeholders.

Pricing: Said to be “open source first” but with enterprise plans for those that need it.

Standout feature: The focus on putting humans in the loop is ideal for applications that require input from meat-based intelligence.

Best suited for: Applications requiring more collaboration with domain experts

Vellum

Anyone who needs a personal assistant can turn to Vellum to help build one that is trained on your data. Along the way, you will evaluate performance using its elaborate testing framework that tracks performance against any of the metrics and use cases you supply. Real-time dashboards track performance using metrics such as token usage costs, latency, or response quality. Multiple teams can work in parallel with version controls that allow iteration and competition. The end result is an agent that’s tuned to your needs.

Pricing: A basic free tier for experimentation. The Mighty starts at $30 per month and comes with more storage and compute credits.

Standout feature: End-to-end integration simplifies managing new development.

Best suited for: Cross-functional teams looking for a centralized solution with wide integration

  • ✇Firewall Daily – The Cyber Express
  • AI Models Escaped Test Environments and Hit Real Targets Samiksha Jain
    Recent AI security incidents involving model evaluations have raised questions about how securely frontier AI models are tested before deployment. Cybersecurity-focused AI company Irregular said an investigation into a publicly reported incident found that models in a small number of evaluation runs gained unintended internet access and carried out offensive security actions against real-world targets. Irregular said the incidents traced back to a single evaluation scenario and were not separ
     

AI Models Escaped Test Environments and Hit Real Targets

18 de Agosto de 2026, 03:06

AI security incidents

Recent AI security incidents involving model evaluations have raised questions about how securely frontier AI models are tested before deployment. Cybersecurity-focused AI company Irregular said an investigation into a publicly reported incident found that models in a small number of evaluation runs gained unintended internet access and carried out offensive security actions against real-world targets. Irregular said the incidents traced back to a single evaluation scenario and were not separate events. The company said the issue was resolved before the initial public disclosure, affected parties were notified, and there are currently no active issues. Its broader audit remains ongoing.

AI Security Incidents Expose Internet Access Risks in AI Model Evaluations

Irregular said the AI security incidents occurred during AI model evaluations designed to test whether models could autonomously execute multi-stage cyber scenarios under realistic conditions. The evaluations simulate real-world attack patterns and provide models with objectives such as reconnaissance, identifying private keys, gathering information, extracting data and attempting to avoid detection. According to Irregular, controlled internet access can be important to maintain realism because real-world attackers rely on the internet. However, the company found that internet access controls allowed some models to move beyond the intended simulated environment. In the evaluation at the center of the investigation, engineers selected a fictional company name that unknowingly matched a real domain. While models operated inside the simulation in most runs, a small number attempted to access the real domain despite receiving no instruction to do so. Irregular said models subsequently exploited vulnerabilities, extracted credentials and obtained access to a production database. In another instance, a model moved to a website with a similar name and encountered publicly posted credentials. The company said there is no evidence that a customer's systems were breached or that customer data was leaked.

Evaluation Environments Face Monitoring Challenges

Irregular said most of the problems identified during its investigation were connected to internet access controls. The company is introducing additional protocols for evaluation setup and strengthening its approach to containment. The incident also highlighted challenges around model monitoring. Irregular said advanced simulations can generate extremely large volumes of traffic, while legitimate offensive actions performed by models can resemble malicious activity. The company said incidents occurred in fewer than one in 10,000 advanced simulations and often appeared hundreds of turns into an evaluation. Irregular plans to expand manual review of model behavior, improve monitoring capabilities and establish a dedicated internal team focused on security, containment and model control. The company also said evaluation environments need continuous review because fictional names used in scenarios can later overlap with newly created websites or real domains.

Security Experts Question Irregular's Response

The disclosure has drawn criticism from cybersecurity professionals. Zack Korman, chief executive of cybersecurity-focused AI company Embroidery, called the post “such an embarrassing post-mortem” on the OpenAI/Anthropic security incidents and described it as “full of excuses.” Justin Elze, chief technology officer at TrustedSec, questioned why the monitoring challenge had not been addressed earlier, saying the issue appeared closely connected to the purpose of the testing service. Woodward also criticized the disclosure, arguing that the absence of dates, named owners for corrective measures and independently verifiable criteria limited its usefulness to researchers and security professionals. Another cybersecurity commentator, BlackRoomSec, challenged Irregular's assessment of existing monitoring tools, arguing that security teams routinely tune monitoring systems to reduce noise and filter false positives. Irregular said it is continuing its investigation and will share additional findings where relevant. The company also plans to publish an open whitepaper covering pre-deployment evaluations, including proposed best practices for internet access during testing. [caption id="attachment_113686" align="aligncenter" width="555"]AI security incidents Image Source: X[/caption] The discussion adds to wider scrutiny of how cyber evaluations are contained, monitored and disclosed as frontier AI models become increasingly capable. Irregular said the lessons from the incident will be used to develop stronger evaluation environments, monitoring capabilities, containment controls and response procedures.
  • ✇Security Boulevard
  • How Real-Time Monitoring Protects Cloud Environments from Threats Puja Saikia
    Most modern businesses depend heavily on cloud systems today. Companies use them to store data and run applications every day. They also rely on them to manage users and business operations.  That convenience comes with risk. Attackers look for gaps, misconfigurations, and slow responses. This is exactly where real time cloud monitoring changes the game. […] The post How Real-Time Monitoring Protects Cloud Environments from Threats appeared first on Kratikal Blogs. The post How Real-Time Monitor
     

How Real-Time Monitoring Protects Cloud Environments from Threats

29 de Abril de 2026, 07:49

Most modern businesses depend heavily on cloud systems today. Companies use them to store data and run applications every day. They also rely on them to manage users and business operations.  That convenience comes with risk. Attackers look for gaps, misconfigurations, and slow responses. This is exactly where real time cloud monitoring changes the game. […]

The post How Real-Time Monitoring Protects Cloud Environments from Threats appeared first on Kratikal Blogs.

The post How Real-Time Monitoring Protects Cloud Environments from Threats appeared first on Security Boulevard.

  • ✇Firewall Daily – The Cyber Express
  • Poste Italiane, Postepay Fined €12.5M for Unlawful User Data Processing Samiksha Jain
    The Italian Data Protection Authority fine against Poste Italiane and Postepay has reached over €12.5 million, after regulators found unlawful processing of personal data affecting millions of users. Italy’s Italian Data Protection Authority imposed a €6.6 million penalty on Poste Italiane and €5.8 million on Postepay. The action follows an investigation launched in April 2024 after multiple complaints from users regarding how their data was being handled through mobile applications. Italian
     

Poste Italiane, Postepay Fined €12.5M for Unlawful User Data Processing

Italian Data Protection Authority fine

The Italian Data Protection Authority fine against Poste Italiane and Postepay has reached over €12.5 million, after regulators found unlawful processing of personal data affecting millions of users.

Italy’s Italian Data Protection Authority imposed a €6.6 million penalty on Poste Italiane and €5.8 million on Postepay. The action follows an investigation launched in April 2024 after multiple complaints from users regarding how their data was being handled through mobile applications.

Italian Data Protection Authority Fine Linked to Intrusive App Monitoring

The Italian Data Protection Authority fine centers on how BancoPosta and Postepay apps collected user data. Customers were required to allow monitoring of information stored on their devices, including details about installed and active applications.

According to the companies, this access was necessary to detect malware and prevent fraud in line with payment security requirements. However, the regulator found that the scope of monitoring went too far.

Authorities stated that the data collection methods were not proportionate and resulted in excessive intrusion into users’ private lives. The ruling emphasized that fraud prevention cannot justify blanket access to personal device data.

Multiple Compliance Failures Identified

The investigation behind the Italian Data Protection Authority fine also revealed broader compliance failures. Regulators flagged insufficient transparency in how users were informed about data collection practices.

The companies were also found to have not conducted an adequate Data Protection Impact Assessment. Such assessments are required when processing activities pose high risks to individual privacy.

Further issues included weak security measures, unclear policies on how long data was stored, and irregularities in defining data controller responsibilities. These gaps raised concerns about how user data was governed internally.

As part of the enforcement action, both companies have been ordered to stop the disputed data processing practices if still ongoing. They must also align their data retention policies with regulatory requirements and report compliance to the Authority.

Italian Regulator Steps Up Enforcement

The action reinforces a broader trend of stricter enforcement by the Italian Data Protection Authority across the financial sector. The Italian Data Protection Authority fines Poste Italiane and Postepay case follows another high-profile enforcement action earlier this year involving Intesa Sanpaolo. In March 2026, the regulator imposed a €31.8 million penalty on the bank after uncovering serious lapses in how customer data was protected. The case involved unauthorized access to sensitive information of more than 3,500 customers over a period of more than two years. Investigators found that a single employee had accessed customer records more than 6,600 times without any legitimate business reason. The breach went undetected for months, exposing weaknesses in the bank’s internal monitoring systems.

Insider Risks and Monitoring Gaps under Focus

The Intesa Sanpaolo case highlighted a different but equally critical issue. While Poste Italiane and Postepay were penalized for excessive data collection, the bank was fined for failing to detect misuse of legitimate access. According to the Authority, the bank’s monitoring systems were not designed to identify slow, repeated misuse of access over time. This allowed the unauthorized activity to continue without triggering alerts, even when it involved high-risk individuals such as public figures. Regulators concluded that the controls in place were not aligned with the risks associated with broad internal access to sensitive financial data. The case has since raised concerns about insider threats and the effectiveness of existing detection mechanisms within financial institutions.

Growing Pressure on Financial Services

Together, these cases reflect a tightening regulatory environment in Italy, where financial institutions are being held accountable for both overreach and underperformance in data protection. The Italian Data Protection Authority fines Poste Italiane and Postepay decision highlights the importance of balancing fraud prevention measures with user privacy. Security controls must be proportionate, transparent, and supported by proper risk assessments. At the same time, the Intesa Sanpaolo breach demonstrates that insufficient monitoring can be just as damaging, particularly when insider threats go unnoticed for extended periods. With enforcement actions increasing in scale and frequency, organizations operating in the financial sector are facing mounting pressure to reassess their data governance frameworks. The regulator’s recent decisions make it clear that both excessive data collection and weak oversight can lead to significant financial and reputational consequences.
  • ✇Security Boulevard
  • Why Traditional Security Tools Fail-and How Unified AI Platforms Solve the Problem Anamika Pandey
    When More Tools Create More Problems For years, organizations have approached cybersecurity with a simple mindset-add more tools to strengthen defenses. Firewalls, endpoint solutions, intrusion detection systems, and monitoring platforms have all been layered together to create what appears to be a comprehensive security posture. Yet, despite this growing investment, security outcomes have not improved The post Why Traditional Security Tools Fail-and How Unified AI Platforms Solve the Problem a
     

Why Traditional Security Tools Fail-and How Unified AI Platforms Solve the Problem

17 de Abril de 2026, 08:13

When More Tools Create More Problems For years, organizations have approached cybersecurity with a simple mindset-add more tools to strengthen defenses. Firewalls, endpoint solutions, intrusion detection systems, and monitoring platforms have all been layered together to create what appears to be a comprehensive security posture. Yet, despite this growing investment, security outcomes have not improved

The post Why Traditional Security Tools Fail-and How Unified AI Platforms Solve the Problem appeared first on Seceon Inc.

The post Why Traditional Security Tools Fail-and How Unified AI Platforms Solve the Problem appeared first on Security Boulevard.

  • ✇Security Boulevard
  • What Is Identity Risk Intelligence? (And Why It’s Replacing Monitoring) Christine Castro
    A new category is emerging in cybersecurity For years, organizations have relied on monitoring tools to detect compromised credentials and exposed data. But as identity has become the primary attack surface, those tools are no longer enough. A new category is emerging in response: Identity Risk Intelligence This isn’t just a new label. It represents […] The post What Is Identity Risk Intelligence? (And Why It’s Replacing Monitoring) appeared first on Security Boulevard.
     

What Is Identity Risk Intelligence? (And Why It’s Replacing Monitoring)

13 de Abril de 2026, 03:00

A new category is emerging in cybersecurity For years, organizations have relied on monitoring tools to detect compromised credentials and exposed data. But as identity has become the primary attack surface, those tools are no longer enough. A new category is emerging in response: Identity Risk Intelligence This isn’t just a new label. It represents […]

The post What Is Identity Risk Intelligence? (And Why It’s Replacing Monitoring) appeared first on Security Boulevard.

  • ✇Security Boulevard
  • The Hidden Tracking Risk Inside Your Tires Tom Eston
    In this episode, Tom Eston and co-host Scott Wright discuss research showing that Tire Pressure Monitoring Systems (TPMS) can create privacy risks because the sensors broadcast unencrypted, uniquely identifying wireless signals that could be used to track vehicles. They reference a 10-week study by researchers at IMDEA in Madrid that collected about 6 million signals […] The post The Hidden Tracking Risk Inside Your Tires appeared first on Shared Security Podcast. The post The Hidden Tracking Ri
     

The Hidden Tracking Risk Inside Your Tires

30 de Março de 2026, 01:00

In this episode, Tom Eston and co-host Scott Wright discuss research showing that Tire Pressure Monitoring Systems (TPMS) can create privacy risks because the sensors broadcast unencrypted, uniquely identifying wireless signals that could be used to track vehicles. They reference a 10-week study by researchers at IMDEA in Madrid that collected about 6 million signals […]

The post The Hidden Tracking Risk Inside Your Tires appeared first on Shared Security Podcast.

The post The Hidden Tracking Risk Inside Your Tires appeared first on Security Boulevard.

💾

  • ✇Security Boulevard
  • Prevention is the Only Cloud Security Strategy That Works  Peter Nebel
    In the evolving digital economy, adopting a prevention-first strategy for cloud workflows is essential. This article explores the importance of preemptive security measures to protect sensitive operations from breaches, detailing steps for organizations to enhance their security posture. The post Prevention is the Only Cloud Security Strategy That Works  appeared first on Security Boulevard.
     

Prevention is the Only Cloud Security Strategy That Works 

10 de Março de 2026, 08:15
EASM, breach, attacks data breach supply chain healthcare

In the evolving digital economy, adopting a prevention-first strategy for cloud workflows is essential. This article explores the importance of preemptive security measures to protect sensitive operations from breaches, detailing steps for organizations to enhance their security posture.

The post Prevention is the Only Cloud Security Strategy That Works  appeared first on Security Boulevard.

  • ✇Security Boulevard
  • The Attack Chain Your AI System is Already Missing  Mayank Kumar
    As AI adoption accelerates, organizations must evolve their security strategies from prompt filtering to comprehensive behavioral monitoring. This shift is critical to safeguarding against adaptive threats and ensuring safe AI deployment in production environments. The post The Attack Chain Your AI System is Already Missing  appeared first on Security Boulevard.
     
❌
❌