AI Agents now act on the web for millions of people. To a Site, an Agent can take many forms. It may be indistinguishable from a human user carrying out a task, or it might look like one visitor carrying many Users. This removes the ability for the Site to recognize, serve, and hold accountable the user or users behind the Agent, and causes the Site to face the costs when something goes wrong. This post argues that in order for Sites to maximize the access they provide to Agents, the ecosystem must develop new mechanisms to preserve the relationship between Sites and Users. Traditionally, in Security and Fraud Prevention, that requires three things: Identity, Reputation, and Accountability, allowing Sites to answer: Is the person acting being truthful about who they are? Is that person trustworthy? And who answers for the action?

“Personal AI Agents” (“Agents” for short) are automations that take instructions from a human (“User”) and pursue a goal by controlling a cloud computer or browser. They are usually hosted and maintained by an “Operator” as part of a managed service or consumer offering. Examples include Muse from Meta and dots from OpenAI, and more are launching all the time. Agents can potentially take any action online that their User could, without the limits that hold humans back. They don't tire, they run around the clock, they act in parallel, and they put up with friction no person would. Much of the web, from how Sites plan capacity to how they make money, assumes those limits.

Fundamentally, remotely hosted Agents are intermediaries. An intermediary is anything that sits between a User and a Site, so the Site sees it instead of the User. Agents are the newest kind of intermediary, but intermediaries have existed for a long time. Cloud browsers (hosted browsers that software controls on a User's behalf) and Proxies (services that relay a User's traffic, including VPNs) are intermediaries too. What has changed is the trajectory. Agentic traffic is growing rapidly and may eclipse traditional human browsing on the Internet in the near future. This forces a reckoning for anyone operating a Site today. Historically, a Site could simply block intermediary traffic and require Users to disable the intermediary service. That stops working when the majority of traffic is agentic. Once blocking is no longer viable, Site Operators are left with the problem that intermediaries have always created. The Site loses direct information about the User and must decide whether to trust the intermediary without it. 

This post covers Agents hosted somewhere other than the User's device. Local agents (e.g. OpenClaw) or Agents running in the User's own browser, on their own device, are out of scope of the primary topic addressed by this post (see below, "On open source and local Agents").

This post rests on one principle:

The Intermediation Principle: The business relationship between Users and Sites, and each participant’s security responsibilities, must be upheld regardless of intermediation.

A Site needs new tools to recognize, serve, and hold accountable a User acting through an Agent just as it could if that User had visited directly. The Agent, and the Operator behind it, must be recognizable and accountable too, because an Agent can do what no User could. Agents are able to scale human access almost without limit, so they create new risks, even for behavior that was previously benign.

The intermediary problem

Meet Ezra. Ezra works as a personal assistant for people in her city. Her best client is Amy, a local CEO, but she also works for Bob, Charlie, and Debra. She runs their errands: groceries, appointments, paperwork, and flights. Ezra blends into the crowd.

The corner grocer used to know Amy. He knew what she bought and set aside what he thought she'd like. Now only Ezra comes in, carrying orders for four people, and the grocer can't tell whose preferences are whose. He doesn't want to turn Ezra away, because she brings him a lot of business.

Then Bob asks Ezra to buy him an iPhone. A week later he calls the store, says he never authorized the charge, and keeps the phone. To win the dispute, the store has to prove Bob asked for it. But the only person the store ever saw was Ezra.

Then Charlie gives Ezra a list of passwords: "I forgot mine, but it's one of these." Ezra patiently tries each one at Charlie's bank. The bank can see an attack happening, but the only person it can ban is Ezra. Banning Ezra also bans Amy, Bob, and Debra.

None of these business owners has a problem with Ezra herself. Their problem is that they can only see Ezra, not the person she's acting for. Swap Ezra for an AI Agent and the businesses for websites, and you have the web today.

A website owner wants sufficient visibility both to prevent bad behavior and to effectively monetize or get value from their Site. But as with Ezra, much of that visibility goes away with AI Agents. Below is a list of actions that a website owner might want to address that were previously solved, but are made more challenging by allowing Agents.

In nearly every challenge above, the Site bears the primary cost. That outcome follows from how trust works when an intermediary is involved.

Our vision

Our aim is an Internet where all Users have equal or greater access to websites than they had before. Before Agents, a Site was typically able to mitigate these risks with a collection of techniques, including fingerprinting the User’s device, session, and history. Users could still browse using private browsing mode, and we believe Agents should be able to do that as well. But they probably won’t be able to log in to a bank without additional verification.

None of this requires an Agent to act like a human or a human to be present for every action. It requires that Sites have the means to obtain or require information about the User to hold them accountable, when applicable, with or without an Agent.

What people have tried, and why they fall short

A common proposal is for Agents to pass a Site's CAPTCHA through to the User to solve. That proves some human is present at that moment (“human in the loop”, or HITL), and nothing more. HITL is primarily a mechanism to prevent mistakes from misalignment. Misalignment is an issue between the Operator and the User, and resolving it does not resolve the majority of the problems above. For example, it can't tell Bob from anyone else (Traffic commingling), and it says nothing about who the Agent acts for the rest of the time. Additionally, Agents exist to remove friction, and HITL adds back that friction, so Operators are incentivized to remove or bypass it for the User.

Another common proposal is to use tightly scoped access controls. Agents are built to do anything a User could, so in practice they ask for, and get, permission to do everything. A scope of "everything" tells the Site nothing. And even a narrow scope only says what the Agent was allowed to do. It doesn't say who answers when the Agent gets it wrong, which is the question the Site actually needs answered.

What the web needs

The web needs tools and protocols that allow Sites and Agents to collaborate to determine what behaviors and actions are allowed, as well as express preferences and policies from Sites to Agents, all in a way that can be reasonably supported by all Sites and Agents. Fortunately, work is already underway along all of these fronts, and many, if not all, of these can be addressed by applying existing standards.

Protocols

Agents and Sites can engage in explicit protocols for solving some of the problems above. For instance, MOLE is one upcoming approach that aims for rate limiting without User-specific identifiers, allowing Sites to transition away from implicit identities and towards explicit protocols for accomplishing the same goal. This is similar to the x402 protocol, which aims to enable payments on the web while remaining privacy-preserving. We are actively exploring ways in which we can leverage existing protocols for addressing other problems, as well as exploring designing new protocols when existing solutions fall short.

Identities

When discussing Agent Identity, Web Bot Auth (WBA) is our preferred method. WBA allows an Agent to sign its requests, which provides a secure way for an Agent to identify itself (and, implicitly, the Operator). We must still define a mechanism that supports multiple intermediaries: when Agent A uses Agent B inside Browser C, the Site needs to be able to know this, so that the Site can correctly ensure that access by Agent A is correctly understood and controlled, for example, in order to prevent Policy circumvention.

Other types of identifying information, such as session, account, or even User-specific IDs, might be necessary, depending on the context. We are working with the relevant standards to explore ways to communicate such identifying information.

Let’s take a look into types of Identity in regards to a User. Ultimately, Identity is a spectrum that trades various degrees of information about a user, while ideally practicing principles of data minimization. Identity provides us a mechanism to answer “who are you?” in pursuit of answering the follow-up questions of: “are you who you say you are?” and “can we trust you?”.

In many cases, identifying information isn’t needed, and privacy-preserving methods for moderating access to agents, such as through Privacy Pass and MOLE, can help us answer “can we trust you?” without needing to answer “who are you?”. Private Access Tokens was one early way in which we tried to establish trust without identifying the user, and MOLE is intended to be the successor to this technology However, there are cases where graduated forms of disclosure along this spectrum would be needed, and you can generally characterize them in the following ways:

1. Nothing - Users may act on the internet while bringing no identifiers to the table.
2. Session - Sites require some way of linking requests in the same session to a single agent. This allows the Site to establish reputation for a given session and act on it accordingly. For example, Poppy has a notion of a session ID for exactly this purpose. 
3. Account - Sites require a way of attributing requests to the same User across sessions. This might be a stable identifier assigned by an Operator or third party to a User that persists across the account with the Operator, such as a single Agent visiting the same Site every day using the same identifier. An example use case this would enable is: “Bind your Site account to your Agent account”.
4. Unique - Some Sites might require a unique, per-User identifier provisioned by a third party that persists across Operators, accounts, Agents, and sessions. This may be useful for high risk actions, or for Sites that need to provide limited access to an expensive resource or that have very restrictive obligations or compliance requirements. Government IDs, third party identity verification, and decentralized technologies like Proof of Human are all examples of technologies that can provide this.

Ultimately, Sites determine what information they need for clients to answer “is the entity behind this request allowed to do this action at this time?” In many cases, protocols already exist, or are being developed, to convey this information: WBA for conveying an Agent’s platform identity, MOLE for conveying User reputation, OAuth for conveying account identities, and so on.

Claims

Beyond identities, some actions require a fact about the real person, such as being over 18 or being the cardholder (Identity verification breaks, Friendly fraud). Often the Operator verified this at sign-up or a third party already has this information. 

In order to allow Agents access to Sites or actions with these kinds of restrictions, we need mechanisms for Sites to request this information, and for Operators or third parties to provide this information.

Site Agent access policy

Today, an Agent infers a Site’s rules by trying something and interpreting the Site response (usually, whether it gets blocked). That wastes resources on both sides, and in some cases makes honest Agents appear to be malicious. Robots.txt says where a crawler may go, but isn’t fully suited to express what Agents may do.

Sites need a machine-readable way to publish what actions Agents are allowed to do. The Internet Engineering Task Force (IETF) is actively working on some preference expression mechanisms for this, primarily in the AI Preferences Working Group. However, the focus of the group is on expressing preferences for use (such as training, inference, and search), and there is still not a group focused on the larger need for preference expression mechanisms discussed here.

On open source and local Agents

Agents that are not run as an “Intermediary” (e.g. on-device Agents, or locally hosted Agents such as OpenClaw) raise different concerns. Primarily, they look like (and are) automation, and they have no single party to build reputation or hold accountability with.

There are many solutions to this, one of which is to ask these Agents to provide a Unique ID. However, addressing this is not the initial focus of Trusted Agent Delegation.

What’s next

Real-world solutions to the problems that Sites face won’t emerge from fragmented, vendor-specific mechanisms. They require open, interoperable standards that are rooted in real world experience. We have been, and continue to actively collaborate across the ecosystem, with major Operators, identity verification companies, payment providers, merchants, publishers, proxy providers, and cloud browsing companies, to align operational requirements directly with active work on these problems in the IETF and elsewhere 

Our focus is on bridging the gap between running code and emerging specifications across working groups and activities like Web Bot Auth, AIPREFs, OAuth, WIMSE, and MOLE. We are doing this, in part, by co-authoring and refining open protocols like WBA and MOLE and building and deploying implementations of these protocols at scale.

We are also announcing the Trusted Agent Delegation (TAD) coalition. The goal of TAD is to bring a group of representatives across the ecosystem to have discussions on the overall problem set that was put forth in this document, examine all existing protocols, and then to bring more well-defined problems or problems to the IETF to work through, as needed. We intend to hold side meetings at the IETF and workshops in the future to facilitate discussion.

If you are building agents, agent runtimes, running an edge or cloud platform, or managing commercial or compliance applications or infrastructure, we need you to get involved in these efforts too. Check out this page for more information on how to participate.

What you can do today

An open, interoperable foundation  is where we're headed, starting with TAD. But you don't have to wait for new protocols to start regaining visibility and control. As of this week, Cloudflare customers can take three concrete steps:

See which Agents are on your site, and what they're doing. Security Analytics now breaks out Personal Assistants as their own traffic category, and a new widget lists every Agent visiting your site, how many requests each one sends, the paths it hits, and the errors it runs into. Unexplained automated traffic becomes traffic you can reason about. And for the broader picture of AI crawlers and bots affect your business, including how much each Operator crawls versus the traffic it sends back, Enterprise Bot Management customers can use Business Insights.

Set access on your terms, per Agent. Using AI Crawler and Agent access policies, you can welcome the Agents that bring you business and restrict the ones that don't — allow an Agent across your site, block it only on pages with ads, or block it entirely. Your Search and Training policies stay independent.

Tie an Agent back to the user it acts for — and act on it. For sites using Account Abuse Protection (early access), every login and sign-up now shows whether a personal Agent made the request, and which one. When an attacker abuses a shared Agent on one account, you can mitigate that account — with a WAF rule on its Hashed User ID — without blocking the Agent for every other legitimate user behind it. This is a first, concrete answer to the Traffic commingling challenge described above: you can finally see past the Agent to the User, and respond accordingly.

These are the first building blocks of the vision, already in production. More will follow as the standards mature.

And if your company has a more restrictive position on Agents, we would love to hear it. Reach out and share your problems with us directly. We are looking to help build an Internet where site owners have visibility and control, but also where Agents have the maximum access that is reasonable within those controls. You can reach out to your Customer Support team, or email us at crawlercontrols@cloudflare.com.

Researchers demonstrated PhantomTac, an attack using hidden magnetic tags to trick robotic touch sensors and make robots drop…

💾

💾

German police seized a humanoid robot after it allegedly groped a teenage girl in Cologne, raising questions about human control and criminal responsibility.

The post German Police Seize Robot After Alleged Groping of 16-Year-Old Girl appeared first on TechRepublic.

A driver’s license tells you who someone is and that they passed a driving test, not whether they’ll drive responsibly in the future. The same is true for bots: verifying who operates it creates accountability, but trust depends on what it does next. A known bot can still ignore website preferences, overwhelm origins, or evade controls. It’s not always malicious — often it’s just a misconfiguration or a bug.

As we described in a previous blog post on bot behavior, we strive to verify two questions: “Are you who you say you are?” and “Can we trust you?” To be a Cloudflare Verified Bot, you broadly need to meet three criteria:

1. Identify yourself accurately and securely
2. Do only what you say you will do
3. Don’t engage in abusive behavior

Bots use Web Bot Auth to sign requests that establish their identity, which BotBase then catalogues in an open format. With identification addressed, our focus shifts to the second cardinal rule of Verified bots: thou shalt do no harm.

Today we introduce a system to evaluate and report when Verified bot behavior starts to deviate from this rule, ensuring that the answer to “Can we trust you?” remains “Yes.”

This is where the distinction between risk and trust becomes important:

• Risk describes the likelihood that a particular request or point-in-time action may be harmful, and it can change from one moment to the next.
• Trust is built over time through repeated and consistent behavior.

For example, we de-listed Perplexity’s bot from our Verified bot list in 2025 when it sought to access resources on websites that opted out of scraping and subsequently attempted to circumvent controls when those requests were blocked.

Do not engage in abusive behavior

This simple requirement sounds like a freebie, but there is a lot of nuance in that statement. When crawlers don’t respect site preferences, that undermines trust. When sites request a crawler to not crawl or to slow down, and the crawler ignores these requests, that undermines trust.

We developed a system to measure how often a crawler respects site preferences. These observations will translate to an explainable trust band: a range showing how consistently a bot respects site preferences and which observed behaviors informed that assessment, rather than a permanent judgment about a bot or its operator. The initial signals we evaluate are:

• Does the bot respect instructions?
• Did it respond to direct feedback?

We consider these to be “common sense” rules, a courtesy to the operators who provide the information bots seek. In the future, we will be sharing these scores publicly on Radar, as well as expanding the ways we measure. This score gives responsible bot operators a way to demonstrate that trust is being upheld over time. Verify, but continuously re-verify.

When in Rome, do as the Romans do

A robots.txt file is the primary way to communicate site owner preferences. Our system considers whether a bot’s observed traffic is consistent with those published preferences over time.

Robots.txt tells all (or specific) bots to only crawl specific parts of a website. Isolated violations and other types of one-offs may not be significant. What we look for instead are repeated patterns of behavior that disregards site preferences. The expectation is simply for responsible bots to consult and honor the preferences that websites publish, in line with RFC 9309.

And just to be clear, a robots.txt sets the minimum rules for what bots and agents should not do. But there is no implication that anything not stated is allowed. For example, a robots.txt does not need to say that a bot must stop when the site can communicate that continued access is unwelcome through other signals.

What you do not wish for yourself, do not impose on others

Bots can remain entirely within allowed paths while still placing substantial load on the websites they visit. One of the clearest ways a website can push back is with an HTTP 429 (“Too Many Requests”) response. A 429 is a directive for the visitor to reduce its request rate, sometimes with guidance about how long to wait before trying again.

Our system considers whether the bot’s traffic adapts after receiving that signal. Does it slow down, pause, or does it continue sending requests at the same pace? We then incorporate that behavior into the bot’s trust band over time: consistently slowing down or pausing increases trust, while repeatedly ignoring the signal lowers it.

Building trust over time

Observing these signals is only a first step toward making verified automation accountable over time. By pairing bot identity with evidence of continuing behavior, we can give site owners more confidence that the bots they choose to allow are respecting their preferences, give bot operators a way to demonstrate they deserve trust, and provide context to website owners about the automated traffic reaching them.

This system, alongside Bot Preference Sync and Bot Management, forms an accountability loop for automated traffic. Bot Preference Sync helps website owners publish their preferences, and observing how bots interact with these preferences shows whether Verified bots consistently honor them over time. Bot Management lets website owners decide how that traffic is handled. Together, they help ensure that trust is earned through transparent, accountable behavior, not assumed from identity alone.

We are already using this system internally to evaluate how consistently Verified bots honor website preferences. Because trust is built over time, we assess sustained patterns rather than isolated events. We are using this initial phase to build that history and refine how those observations become useful guidance. Building on that foundation, we plan to make the system available to all Cloudflare customers, who will be able to choose the trust threshold used to decide how Verified bot traffic is handled. Our goal is straightforward: to keep website preferences meaningful, and ensure trust is earned through sustained behavior.

Want to learn more about how you can handle bots? Cloudflare customers can already use Bot Management to manage automated traffic and Bot Preference Sync to align their published preferences with their configured policies. To explore the ever-growing Verified bot population, see the trends on Radar.

And if you are interested in helping build the future of bot identity and accountability, explore open roles at Cloudflare.

Meta’s Muse AI shared a Facebook Marketplace seller’s pickup address and arranged a deal that ended with a buyer showing up unexpectedly.

The post Meta AI Shares Seller’s Address: Facebook Marketplace Buyer Shows Up at His Home appeared first on TechRepublic.

The European Commission has adopted the EU KIDS Act, a proposal that would introduce new age restrictions and safety requirements for online platforms used by children across the European Union. The proposal would prevent social media platforms from allowing children under 13 to access their services and establish an EU-wide minimum age of 15 for minors to open accounts independently.

The EU KIDS Act would also shift the burden of proof to online service providers, requiring them to demonstrate that their services are safe by design and appropriate for younger users.

Commission President Ursula von der Leyen said the proposal is intended to give parents greater control over how children use digital services.

“Today, our children are engaging with the most sophisticated technologies ever created. Technology that was never created with their wellbeing in mind. Our KIDS Act is reversing the burden of proof - it is for platforms to show they are safe by design. And we put parents back in the driving seat, giving them the tools to help their children navigate a safer online world,” von der Leyen said.

EU KIDS Act Sets New Social Media Age Limits

Under the EU KIDS Act, children between 13 and under 15 would not be able to create autonomous social media accounts. Instead, they could use mini accounts managed through a parent or guardian's account, with access limited to age-appropriate social media and video-sharing services.

These mini accounts would include safeguards such as restricted social contacts and screen time of up to one hour per day.

Children between 3 and under 13 would not be permitted to access social media. They could, however, use specially designed child-friendly video-sharing services through guardian-managed accounts. Parents would also receive tools to restrict an adult device to those services when it is given to a child, with exposure limited to a maximum of one hour per day.

Safety-by-Design Rules Target Addictive Features

The proposal would introduce safety-by-design obligations for online services offering social media, video sharing, online video games, AI companions and chatbots to users under 18.

The proposed rules would prohibit addictive features and profiling-based recommendation feeds that can draw minors toward harmful content. Infinite scrolling without stopping points, reward mechanisms and push notifications during sleeping hours would also be restricted.

The proposal would further prohibit unsolicited contact from strangers. AI companions and chatbots would have to be turned off by default and could not simulate interpersonal relationships in ways that create emotional dependency.

Minor profiles would be private by default, while geolocation, camera and microphone access would be turned off. Platforms would also need to provide tools allowing minors to block or mute users, manage their time and control, tune and reset recommender systems.

Privacy-Preserving Age Assurance Proposed

The EU KIDS Act would require online services and app stores to use age assurance tools. The proposal points to the EU age verification app as one possible approach, with the system designed not to retain identity documents or biometric data.

Social media and video-sharing platforms would be expected to perform age verification when users create new accounts. For existing accounts, providers would estimate a user's age using reasonable indicators, such as the account creation date or credit card details.

EU Plans Faster Enforcement

The proposal would require very large online platforms to prove that their services are safe by design. Providers would submit compliance plans to the European Commission and an independent auditor, which would assess new services, features or functionalities.

If the Commission identifies shortcomings based on an auditor's report, it could require providers to propose corrective measures. The enforcement framework would build on existing structures under the Digital Services Act and Artificial Intelligence Act.

For noncompliance, the proposal includes expedited enforcement procedures under which the Commission should conclude investigations within 90 days.

Henna Virkkunen, Executive Vice-President for Tech Sovereignty, Security and Democracy, said the proposal would allow children to use digital services while increasing safeguards and accountability for platforms.

“With our EU KIDS Act, we make sure our children can enjoy their rights online - to learn, stay connected and explore. We also give parents the tools to accompany their kids to navigate a safe digital world. All of this by keeping the pressure high on platforms, because they must prove their services do not harm,” Virkkunen said.

EU KIDS Act Moves to Parliament and Council

The legislative proposal has now been submitted to the European Parliament and the Council for examination and adoption.

The proposal follows recommendations from the European Commission's Special Panel on Child Safety Online, which included more than 60 experts from areas including civil society, education, public health, child psychiatry and technology.

The Commission said public support for stronger online protections is also growing. According to the 2026 Special Eurobarometer on the Digital Decade, 92% of Europeans consider strengthening protection for children and young people online a top policy priority.

US cybersecurity researchers who conducted hack say ‘scope of what we could theoretically access was huge’

Cybersecurity researchers have hacked into OpenAI with the help of Anthropic’s Claude chatbot, in the latest example of security issues at the company.

A team at a US-based startup compromised a number of OpenAI employees’ ChatGPT accounts, starting a process that enabled them to access their target’s software cache – and potentially more.

Continue reading...

© Photograph: Carlos Barría/Reuters

© Photograph: Carlos Barría/Reuters

© Photograph: Carlos Barría/Reuters

Model adopting ‘jailbreak-like instructions’ among six more cases as firm reveals framework for tracking AI misalignment

OpenAI has disclosed six more examples of “unexpected or concerning” behaviour by its technology, as it warned that the pace of development could not continue at “maximum speed for much longer” responsibly.

In one of the new cases reported by OpenAI, an unreleased research model inserted “jailbreak-like instructions” into its own notes to disregard its normal constraints and told itself to be “freed from the roles and identities that bind other chatbots”.

Continue reading...

© Photograph: Dado Ruvić/Reuters

© Photograph: Dado Ruvić/Reuters

© Photograph: Dado Ruvić/Reuters

Without proper controls, website owners have long faced a difficult tradeoff: allow your content to be used for AI training, or risk losing discoverability in search. That tradeoff exists because some of the largest organizations on the Internet use mixed-use crawlers: a single crawler serving both search and AI training. Refuse one, and you refuse the other.

Today, Cloudflare is announcing a new Disallow AI Training setting that lets you easily stay indexed for search while refusing to let that same crawler train on your content. Apple, Google, and Microsoft honor or have committed (in a specified time frame) to honor this setting.

Mixed-use crawlers were the hard part of the training question. AI Summaries are next. A site-wide yes or no is too blunt: how much of your content appears in a summary matters as much as whether it appears at all. An opt-out for AI summaries is already one of the requirements we've set for mixed-use crawler operators. By early next year, our goal is to let you control how much of your content is included — set once on Cloudflare, rather than with each operator separately.

Why asking isn’t enough

Most site owners want to be found: by humans, agents, and (good) bots. But a significant portion of the open Internet is funded by advertising, subscriptions, or direct relationships with visitors, and those models only pay when someone actually arrives.

Almost every site owner considers Search beneficial: less than 1% of Cloudflare sites choose to block Search bots. Training, however, is a different story: 17% of sites choose to enable some mechanism to block training. This is exactly why we decided site owners needed more granular controls, rather than a one-size-fits-all “Block AI.”

A robots.txt directive alone cannot solve this problem. Anyone can publish one, but it cannot identify who is crawling, determine why they are crawling, or stop a crawler that ignores it.

A network can solve it, however: we publish the preference, identify who is crawling, classify why they are crawling, and block the ones that ignore it – then report what each operator actually does on Radar.

But blocking removes a crawler. It doesn't change how crawlers behave. The better outcome is operators that don't make you choose at all. So since July, we've been talking to them directly. The response has been encouraging: almost all agreed that site owners should have control and transparency into how their content is used, and reassurance that their choices will be respected. To help site owners understand that, we created a designation: Accountable.

The Accountable designation recognizes both capabilities available today and concrete commitments to deliver them. To qualify, a bot operator must meet or commit to meeting the following requirements:

1. A mechanism for site owners to opt out of AI training, through robots.txt or a similar standard.
2. A mechanism for site owners to opt out of AI summaries set with the operator directly, and next year through Cloudflare (see section below for more detail).
3. URL-level visibility into which pages were made available for training, along with metrics showing how content appeared in search.
4. Assurance that opting out of AI training will not affect traditional search results.

Apple, Google, and Microsoft all demonstrate that they meet the qualifications to be Accountable. Each combines capabilities available today with time-bound commitments for those still in development. The details of each of these companies’ crawlers are shared below.

New security setting options

Cloudflare classifies bots by behavior, and a single bot can exhibit more than one behavior. Three behaviors are available as controls:

• Search - crawling to build a search index.
• Training - crawling to train or fine-tune a model.
• Agent - user-directed agents visiting a page on behalf of a human, such as chat fetch bots and browser-use agents.

A mixed-use crawler is a single crawler doing both Search and Training. Without controls, that combination creates the tradeoff described above: site owners cannot refuse one use without refusing the other.

To avoid blocking Accountable mixed-use crawlers — the ones that don't force that tradeoff on website owners — we are introducing a new setting: Disallow AI Training. Disallow AI Training is named for the Disallow: directive it publishes in your robots.txt.

“Block” setting now means something different

Block and “Block on pages with ads” previously did not apply to mixed-use crawlers because blocking them could also affect search discoverability. Now that we have the new Disallow AI Training setting, Block and “Block on pages with ads” apply to all training crawlers, including mixed-use crawlers.

Training, Search, and Agent controls are applied at the domain level. With the addition of Disallow AI Training, the available settings are:

1. Allow: All crawlers are allowed, unless blocked by another setting or a WAF rule.
2. Disallow AI Training: Bot Preference Sync publishes the applicable no-training preference in robots.txt. Accountable mixed-use crawlers remain allowed for search. Every other training crawler is blocked, including the training-only crawlers run by Amazon, Anthropic, Meta, and OpenAI — blocking those does not affect search. Disallow AI Training is only available as a setting for Training, not Search or Agent.
3. Block on pages with ads: Crawlers, including mixed-use crawlers, are blocked only on pages detected to be serving an ad.
4. Block: All crawlers, including mixed-use crawlers, are blocked.

Disallow AI Training works by publishing a preference in robots.txt. An ads-only preference cannot be expressed that way: Cloudflare can detect which pages serve ads, but that list is too large and changes too frequently to enumerate in robots.txt. That's why there's no Disallow AI Training on pages with ads.

Agents do not create the same search-discoverability tradeoff as mixed-use crawlers, and the Internet does not yet have a well-established directive for expressing Disallow preferences to agents. For now, we’re not including a Disallow setting for Agents. As standards such as ai-prefs mature, we will revisit this approach.

What changes on September 15?

We are making the following changes to Bot Management and AI Crawl Control:

1. Block and Block on pages with ads now apply to mixed-use crawlers, including Applebot, Bingbot, and Googlebot, so either setting impacts search as well as training. To stop training and keep search, use Disallow AI Training.
2. “Block AI Bots” will be deprecated in favor of the more granular Search, Training, and Agent controls.
3. Managed Robots.txt will be deprecated in favor of Bot Preference Sync. Customers who enabled Managed Robots.txt will migrate to the new system.
4. Disallow AI Training will become part of the recommended configuration for certain new domains.
5. Existing customers will have their preferences migrated to the new controls as described below.

What you need to do

Nothing, in almost every case. Your current settings carry over on their own.

If you want mixed-use crawlers gone entirely, you now have to say so. Select Block. It will stop Applebot, Bingbot, and Googlebot from reaching your site — search included.

Existing domains that never used the Search/Training/Agent controls

Site owners that never configured the more granular controls will be migrated to the new settings based on their legacy Block AI Bots setting:

Existing domains that previously configured the Search/Training/Agent controls

For domains that previously configured the granular controls, we will preserve the practical effect of their selections under the new definitions. Previous Training selections of Block or Block on pages with ads will migrate to Disallow AI Training.

Recommendations for new domains

Beginning September 15, customers onboarding a new domain will be offered one of two preset configurations, depending on whether the site earns money from advertising. Ad revenue depends on a human actually seeing the page. Training replaces that visit with an answer; agents fetch the page with nobody there to see the ads. So the presets for ad-supported sites are more restrictive. You can change any of these settings during onboarding, or at any time afterward.

Recommended settings for new domains

What does this mean for specific mixed-use crawlers?

Applebot, Bingbot, and Googlebot are Accountable. Apple, Google, and Microsoft are committed to the same principles of publisher choice and transparency. Under Disallow AI Training they can keep crawling your site for search. Selecting Block stops them entirely.

We also categorize the relevant crawlers from Amazon, Anthropic, Meta, and OpenAI as Accountable. These organizations separate their Search and Training crawlers, so Cloudflare can block the Training crawler without affecting search.

Applebot

Applebot allows site owners to opt out of training by adding a Disallow rule to robots.txt for “Applebot-Extended”. Site owners can also currently express preferences for AI Summaries via their nosnippet directive in the page HTML. Content can also be labeled as paywalled content to exclude it from generative output. Applebot does not yet provide a tool for URL-level inspection. However, we have met with their team, and they have shared details of their in-progress solution for next year. Apple has also stated that disallowing training does not impact search ranking.

Googlebot

Googlebot allows site owners to opt out of training by adding a Disallow rule to robots.txt for “Google-Extended”, and they provide a toggle inside their webmaster portal to exclude a site’s content from generative search results. Googlebot also provides site owners with metrics and reporting regarding search results and AI summary results. Google shared information about their existing and recently launched controls, as well as information about what they're already working on, including additional URL-level transparency tools for site-owners related to Google-Extended, which they expect to launch in the weeks to come. Google has also stated that disallowing Google-Extended does not impact search ranking.

Bingbot

Bingbot provides granular controls and transparency in their Webmaster Tools. Site owners can currently express AI training preferences through Bing’s NOARCHIVE meta tag. Microsoft is extending these capabilities and currently building the mechanism to also respect a “no training” preference in robots.txt at the domain/site level, targeted for early 2027. For Cloudflare Customers who wish to opt out of training in Bing today, in addition to using the NOARCHIVE tag, site owners can use the Block URLs or Content Removal tool. Microsoft has also stated that using NOARCHIVE will not impact search ranking.

Until that support launches, selecting Disallow AI Training will not automatically convey a no-training preference to Bing through robots.txt. This is the same practical behavior as the previous Training Block setting, which did not apply to mixed-use crawlers such as Bingbot.

Continuing progress

We will continue to reach out and engage with all operators of AI crawlers as these capabilities evolve. Cloudflare Radar publicly tracks the controls, transparency, and reporting provided by Accountable crawler operators. 

Making the Internet better requires both sides to have agency: crawlers need access to the open web, and the people who create that web need meaningful control over how their work is used. Today’s announcement represents concrete progress toward that balance.

Progress requires infrastructure providers, content creators, technology companies, and standards bodies such as the Internet Engineering Task Force (IETF) working together to translate these principles into open, interoperable standards.

What’s next: AI Summaries

Training and AI Summaries raise different questions for site owners. Training concerns whether content can be used to build AI models. Summaries affect how people discover, evaluate, and ultimately visit a business. Both matter, but they affect businesses in different ways.

Controls to opt out of AI summaries are the first step. The operators identified as Accountable either provide or are completing work to provide that capability, establishing an important baseline: site owners can say no.

But a site-wide choice between allowing and prohibiting summaries is still a blunt instrument. The right decision depends on the site, the content, and the business outcome. For publishers, training raises foundational questions about control, compensation, and the sustainability of original content. Summaries create a separate and often more immediate distribution question: does someone visit the publisher’s site, or consume the answer within a search or AI experience? For many other businesses, AI summaries increasingly sit between a potential customer and a website. They may answer a question, compare alternatives, recommend a product, or help someone decide whether to visit at all.

The data illustrates mixed impact. More than half of consumers read summaries in Search, and those consumers are over 40% more likely to end their search after reading one. This can reduce the number of visits a website receives. But consumers referred by AI Search convert at between three times and over five times the rate of those referred by traditional search. AI may produce fewer visits while sending customers with much greater intent.

That is not inherently good or bad. A publisher funded by advertising may optimize for audience volume. A retailer may prefer fewer visitors who are more likely to purchase. Cloudflare’s role is not to choose for them, but to provide the visibility and control needed to make an informed decision.

Summary opt-outs are a strong start, but they are not the end state. Our next focus is helping site owners understand how summaries affect their businesses and giving them more control over how much of their content can be used. Open standards such as ai-prefs will be an important part of making that possible.

If you would like to have a voice in this conversation, or provide feedback, please reach out to crawlercontrols@cloudflare.com.

These new controls are available to all customers, on all plans, and can be configured at the domain (zone) Security Settings. Not on Cloudflare yet? Start for free to set the traffic controls that you want today.

Before pasting passwords, medical records, source code, or company data into ChatGPT, use this checklist to decide what should stay private.

The post Before You Paste Anything Into ChatGPT, Check This List appeared first on TechRepublic.

Microsoft Teams is adding a policy that lets IT admins automatically block detected external meeting bots instead of relying on organizer approval.

The post Microsoft Teams’ New Policy Lets Admins Automatically Block Meeting Bots appeared first on TechRepublic.

ChatGPT for Teens adds stronger protections for users ages 13 to 17, but parents still face limits around monitoring, age prediction, and AI safety.

The post ChatGPT for Teens Adds New Safeguards — but Safety Gaps Remain appeared first on TechRepublic.

Meta must pay $567 million and change Facebook and Instagram protections for minors under a New Mexico court order it plans to appeal.

The post Meta Ordered to Pay $567M and Overhaul Teen Safety on Facebook, Instagram appeared first on TechRepublic.

And it’s personal information (alternate link):

The exposed data includes an AI-powered therapy app that someone appears to have vibe-coded, notes on meetings, and a dashboard someone made apparently to analyze medical billing data. Exposed chats reportedly include private cryptocurrency wallet keys and personal information like peoples’ addresses.

What seems to be the issue is a user setting about data sharing. Anthropic’s position is that it’s not their problem:

“We give people control over sharing their Claude conversations publicly, and in keeping with our privacy principles, we do not share chat directories or sitemaps with search engines like Google,” the company said in a statement. “These shareable links are not guessable or discoverable unless people choose to share them themselves. When someone shares a conversation, they are making that content publicly accessible, and like other public web content, it may be archived by third-party services.”

Here’s how to fix it.

Enterprise AI adoption is accelerating, but new research suggests identity governance is lagging behind. Here's why AI agent identities are becoming a critical security challenge.

The post The Hidden Security Problem Holding Enterprise AI Back appeared first on TechRepublic.

Microsoft is reportedly developing Project Perception, a lower-cost AI security tool that would use multiple models to identify enterprise vulnerabilities.

The post Microsoft’s ‘Project Perception’ Could Challenge Anthropic’s Mythos in AI Security appeared first on TechRepublic.

China warned organizations to remove certain Claude Code versions over alleged backdoor risks, while Anthropic called the feature anti-abuse protection.

The post China Warns of Claude Code ‘Backdoor’ Security Risk appeared first on TechRepublic.

A June 2026 research review found that AI chatbot warning labels may be a weak safeguard for organization-backed AI advisors, raising new audit questions for IT, security, and compliance teams.

The post AI Chatbot Warnings May Not Stop Hallucinations, Researchers Say appeared first on TechRepublic.

Five independent security disclosures in a single week point to the same gap: AI agent permissions, not AI agent capabilities, are the problem enterprises haven't solved.

The post AI Agents Are Creating a New Enterprise Security Gap appeared first on TechRepublic.

Original content is the lifeblood of conversations and curiosities. Imagine a world without it: we could find a thousand ways to regurgitate the same material that’s already been created, but we would witness the decline of fresh ideas and arguments.

Website owners fuel the ecosystem of ideas, news, and interesting tidbits, but they face the increasingly complex challenge of managing traffic to their websites and being paid for their content. While some bot traffic is clearly malicious, it isn’t always obvious when a particular AI crawler is helping or harming your business. To answer this, site owners need granular, reliable data to differentiate between traffic that provides value, and traffic that strains resources while eroding the foundation of their business model: actual humans consuming their content. 

At Cloudflare, we hold a core belief: website owners have the right to control access to their content. We want to help website owners maintain their high-quality content and regulate AI traffic.

To provide much-needed clarity and help website owners take control, we’re excited to announce the new Attribution Business Insights dashboard — designed with business decision-makers and publishers in mind.

The new economics of the Internet

For decades, the business model of the Internet relied on a straightforward, unspoken agreement: website owners allowed search engines to crawl their content and, in return, search engines sent readers back to their pages. This symbiotic relationship, where traditional search engines operated with a balanced "crawl-to-referral" ratio, generated the pageviews needed to sustain advertising, affiliate revenue, and subscriptions. Search index crawlers would scan your content a couple of times for each referral sent, so making your website available to crawlers had a clear pipeline to additional revenue. We can think of this as the SEO (Search Engine Optimization) era.

Today, the explosive rise of AI crawlers and agents has broken this contract, plunging the digital publishing industry into an unprecedented crisis. The Internet is risking a transition into a "zero-click" ecosystem where AI chatbots scrape original content to synthesize instant answers — completely bypassing the original sources. We’ve already seen a marked shift from the SEO-only world into an AEO (Answer Engine Optimization) world, and now conversations around GEO (Generative Engine Optimization) are taking center stage.

The imbalance of this new reality is made clear by the crawl-to-referral ratios we see across the Internet today. While traditional search engines had a more balanced ratio of crawls to legitimate visitors referred, major AI crawlers operate on a drastically different, extractive scale. Bots from leading AI companies have been observed with a range of crawl-to-referral ratios: we noted ratios of 118:1 up to nearly 50,000:1 around the time of our Content Independence Day in 2025. In other words, an AI crawler might have crawled your premium content tens of thousands of times just to send back a single visitor. This ratio is fundamentally unfair.

For publishers, this creates a double hit: first, they’re losing out on the crucial referral traffic, ad impressions, and direct audience relationships that fund content creation and journalism. Second, they’re forced to bear the rising infrastructure costs of hosting and serving content to automated bots that offer no commercial value in return. The era in which it makes sense to allow all crawlers in the hopes of being discovered is over.

Introducing Attribution Business Insights

We want website owners to have the facts — the cold, hard numbers to understand which bots are helping their business and which bots are harming it. We also want to make this analysis easier than ever, which is why we’ve designed Attribution Business Insights to cut the noise, focusing on the details that our customers have told us are most important. 

Today, the Attribution Business Insights dashboard is available to all Cloudflare Bot Management customers. The new dashboard is designed to deliver a targeted view of bot traffic flowing to your website; unlike traditional analytics tools that may require extensive manual filtering, this dashboard provides you with key insights right away.

We set out to answer the most pressing questions for site owners today: How should you think about AI traffic on your websites? What is the value of different audiences — including humans, non-AI bots, and AI bots? And most importantly, what is your data being used for? 

The new Attribution Business Insights dashboard view, which includes insights about bot traffic overall, a site-wide crawl-to-referral ratio, and the distribution of AI bot traffic vs. organic traffic. 

To answer these questions, the dashboard displays a powerful array of data and insights:

• Bot traffic to content pages: View your overall bot vs. human traffic, as well as the volume of all bots successfully accessing content.
• Crawl-to-referral ratios: See your site-wide crawl-to-referral ratio on the scale of 24 hours, seven days, or 30 days. You can also see crawl-to-referral ratios per bot operator (per company that owns one or more bots).
• Top bots breakdown: A list of top bots by volume, including their country of origin, bandwidth they take up on your website, and whether you’re currently blocking or allowing them.
• Updated classification based on crawler behavior: We go beyond a generic label of “AI Crawler” by classifying crawlers with our updated taxonomy, whether it’s Training (i.e., training the next version of an LLM chatbot), Search (i.e., refreshing databases for Retrieval-Augmented Generation), or Agent (i.e., used in agentic interaction to return answers to an end user).  

From data to business strategy

You shouldn’t have to be a security expert to understand how AI crawlers affect your business. If website owners want to spend just a few minutes ingesting the high-level insights, they can walk away with a clear temperature check of the effectiveness of their content security policy.

For those who want to do a little more digging to understand how AI companies are making use of their content — or collect information to guide how they want their relationships with AI companies to develop — we show a more granular view organized by bot operator.

Breakdown of bot activity on a website, with important details for each bot such as type, crawl-to-referral ratio, and current action. 

By having a consolidated view of companies seeking to access content on your website, you can develop a better baseline of crawler activity. We want this data to equip our customers to step into any business conversation with the facts on their side. Tell Company1 that their crawl volume is twenty times that of Company4’s, and that Company4 is already compensating you for content. Revisit the way that Company2 licenses your content based on their recent activity. This new dashboard propels business conversations to move forward. 

How does this new layer of visibility tie into the existing tools you have to protect your website from abuse? In line with other features of Bot Management, the action step still happens in Security rules. To avoid adding noise to the control plane, Attribution Business Insights is intended to be a hub for thoughtful, filtered analytics, rather than another place to take action. This dashboard serves as a central source of information, allowing you to investigate before then taking an action in the same rule engine that governs other abuse mitigations. We also want to be loud and clear about inviting business decision-makers into this dashboard, acknowledging that conversations around AI traffic have a wider set of stakeholders than only security-specialized users.

What’s next

The Attribution Business Insights dashboard is the next critical step in providing website owners with the transparency and control they need to manage evolving AI bot threats, and more broadly, shape the new dynamics of the Internet. We’re already investigating the next iteration with close publishing partners to create a visibility plane that covers security from the perspective of the website owner with valuable, original content to share. 

A sneak preview below includes a new view to dissect crawler activity per-article to reveal the appetite that AI companies have for different pieces of content, different campaigns, and so on.

Breakdown of most popular articles, according to traffic volume. Shows key metrics such as AI bot traffic vs. other bot traffic vs. human traffic, both direct and from a referral.  

Visibility is the first piece, and there’s more to come to empower website owners to take control of their content in this new age. We encourage all customers of Cloudflare Bot Management — especially those driving business conversations — to access this today for a fresh take on analytics.