On October 19, 2026, we learned about eMudhra’s eight shockingly weak keys: where eMudhra was trusted to pick a 115-digit number for their virtual combination lock protecting the Internet, they picked the number 4. Not even four digits, just the number 4. The other seven keys were similarly weak: 56, 66, 115, 155, 156, 230 and 237.

The next day, we wrote a blog post on the immediate action we took to mitigate this leak, and what customers needed to do on their end to stay secure. Now, we are sharing a more in-depth follow up on the evidence we’ve seen of exploitation, and the progress seen on visitors patching their systems.

Let's start with the scale of the problem. A certificate authority (CA) with weak keys this serious is unprecedented, and the potential consequences are significant. While there have been plenty of compromised certification authorities in the past (such as DigiNotar, TÜRKTRUST, and TrustWave), access to their keys has always been contained to a degree; these incidents gave a limited set of attackers or organizations the ability to issue fraudulent certificates. The eMudhra incident is fundamentally different; the affected intermediate private keys could be recovered almost instantly by anyone from the public certificates themselves. Anyone can independently recover these same signing keys and create certificates for any domain that an unpatched client not enforcing Certificate Transparency would accept.

eMudhra announced they’re exiting the public CA business and have claimed a mistake in a key generation script, but have not yet shared the script. They signed a certificate revocation list (CRL) with what might’ve been a correctly generated key, and subsequently corrected it to be signed with the weak key.

How did that come to be and why didn’t that set off alarms? Much has to be cleared up, including the role of the auditors and cross signers. This blog proposes the fixes and next steps for organizations to ensure they are secure.

Patching Cloudflare

Our primary concern remains the security impact on Cloudflare, our customers, and the Internet as a whole. At the time of writing, all high priority Cloudflare systems have been patched. A small number of confirmed low impact tickets are being worked on.

We started patching high-impact systems immediately on October 19. These included customer origin-facing proxies using Pingora, responsible for most of our outbound connections. These were patched in hours to reject the vulnerable eMudhra keys. At the same time, we started an AI-assisted analysis of our source code to find all other systems that trust these weak keys. Within four hours we had a full analysis. We assigned a ticket with preliminary analysis for each system, and asked the team to confirm and/or adjust the analysis before the end of the day. Where high impact was identified, we asked to patch immediately, and for the lower impact, remediation was due at the end of last week.

Has Cloudflare seen exploitation across its network?

To track whenever a customer’s server (or attacker) presents a certificate signed by eMudhra’s weak keys, we added logging to Pingora. As this outbound service accounts for the vast majority of outgoing TLS connections from a Cloudflare machine, any attacker attempting to snoop using the eMudhra keys would trigger this logging. The logging finished rolling out October 21.

At the time of writing, a week later, we’ve seen about 22,000 attempted handshakes with the weak eMudhra keys. Two-thirds of the attempts were fetches of test sites such as emudhra.we-have-a-problem.com (which we mentioned in the developer guidance in our earlier post). Almost all of the remaining third went to origins who appeared to set up a certificate for themselves using the weak intermediate. Every handshake configured to verify the certificate was refused, as we already patched.

About 160 of these handshakes stood out: the certificates’ Subject Alternative Names (SANs) were naming other organizations’ domains, unrelated to the Server Name Indication (SNI) requested in the handshake. Some of those names were domains related to consumer IoT devices. We noticed as well that those forged certificates were made after the keys leaked and some were valid for over two years, longer than any public CA may issue. However, these certificates looked like the servers' own default certificates rather than part of an attempt to intercept our connections. They named domains other than the ones being connected to, so they would have failed hostname verification anyway. And in several cases, a server presented the same certificate to connections from many of our locations, which means it came from the server itself rather than from something on a single network path.

Thus while there is no evidence of the weak keys being used to intercept Cloudflare's connections so far, these keys appear to have been used to forge certificates within hours of the weakness becoming public: one of the forged certificates we saw was dated October 20 at 18:19 UTC, less than an hour after Mozilla's bug report was made public. This does not exclude carefully targeted interception. For that we have to rely on the usual indirect signals of system compromise — and so far, we have not seen any such signals.

How many visitors are still vulnerable?

So much about patching Cloudflare, but what about our visitors? We cannot directly test whether a visitor still trusts the weak keys without breaking connections, so we have to do something more indirect. For that we used our challenge pages:

We rolled out an update to send out two probe connections on a fraction of challenge pages: one probe to ca1.cloudflarecaptcha.com and one to ca2.cloudflarecaptcha.com. Initially we installed an eMudhra signed certificate on ca1.cloudflarecaptcha.com and a regularly signed certificate on ca2.cloudflarecaptcha.com as control. Then we record whether these probes succeeded. This allows us to see if visitors that are served a challenge page still trust the weak eMudhra keys.

What did we find? With 50 million probes over the last week, we see this:

Big caveat upfront: by design the traffic to these challenge pages is highly skewed. There are many (unconfirmed) bots in there. Also, only clients that execute JavaScript (like browsers) will send probes.

The jump in the data is due to the following changes—all times are UTC:

• October 23 15:30, we changed our control endpoint to also serve an eMudhra certificate, but this time without signed certificate timestamps (SCTs). There were some differences in acceptance, but not large. We could’ve added a new endpoint, but that would’ve required a slow release of our challenge platform. Changing the endpoint was easier.
• October 23 16:30, our challenge platform rolled back to a previous version because of an unrelated issue. The probes returned October 27 at 10:00.
• October 27 16:37, we noticed we did not send the G3 cross-sign, which not all clients knew about, and added it to both endpoints. We removed SCTs from both.
• October 27 16:49, we switched ca2 to chain up to the AAA Certificate Services root.
• October 27 18:18, we ramped up probes to be included from 1% to 10% of all challenge pages served on non-paying customer zones.

With those caveats out of the way: what do we notice? Up-to-date modern browsers are mostly patched, but bots and old browsers are not. Also embedded browsers, like Android WebView, lag behind patching with 30–40% still accepting. Over the first weekend (with the monitoring gap), we see clear evidence of patching, but afterwards we see little movement.

On October 29 around 18:00 UTC, we released a detection that caught a large influx of bot traffic identifying as Chrome 150-154 on macOS. This can look like patching in the top graph, but it is actually a change in traffic classification: the same user agents show a corresponding increase in acceptance among traffic with bot signals detected in the bottom graph.

Chrome 150-154 on Linux is the notable exception: its acceptance continues to decline, consistent with further patching in that cohort. Given the bot-heavy and changing traffic mix, that evidence is suggestive rather than conclusive. Otherwise, we still see little movement.

Breaking down by network (ASN) we see:

Across the large networks, more than half of probes still accept the compromised chain, both among traffic with no bot signals detected (top graph) and traffic with bot signals detected (bottom graph). Over the weekend we saw some patching, but afterwards the trend has been flat.

The big caveat still holds: traffic is skewed by (unconfirmed) bots. But bots are also just software that need security updates. And if they’re in any way representative of overall software updates, then the picture is quite dire indeed.

What can you do?

Patch! Dennis Jackson has a great crowdsourced website with instructions. Especially if you’re a developer on macOS, you want to follow these instructions. If you’re a Cloudflare customer, check out last week’s blog post on what needs to be done on your end when using certain products like WARP or Tunnel.

Looking ahead

At Cloudflare, we have a quick-revocation mechanism in place in some products. To bring it to more systems at Cloudflare and industry, we’re excited about projects like upki that understand a trust store can’t be a static list of root certificate authorities, but needs the ability to be quickly updated with information like revocation and certificate transparency keys. As vulnerabilities are discovered at a faster pace than ever, it is vital that they can be addressed quickly and effectively.

On April 29, 2026, a Linux kernel local privilege escalation vulnerability was publicly disclosed under the name "Copy Fail" (CVE-2026-31431). Cloudflare’s Security and Engineering teams began assessing the vulnerability as soon as it was disclosed. We reviewed the exploit technique, evaluated exposure across our infrastructure, and validated that our existing behavioral detections could identify the exploit pattern within minutes. 

There was no impact to the Cloudflare environment, no customer data was at risk, and no services were disrupted at any point. Read on to learn how our preparedness paid off. 

Background

Our Linux kernel release process

Cloudflare operates a global Linux server infrastructure at an immense scale, with datacenters located across 330 cities. We maintain a custom Linux kernel build based on the community's Long-Term Support (LTS) versions to manage updates effectively at this volume. At any given time, we may utilize multiple LTS versions from various series, such as 6.12 or 6.18, which benefit from extended update periods.

The community regularly merges and releases security and stability updates which trigger an automated job to generate a new internal kernel build approximately every week. These builds undergo testing in our staging data centers to ensure stability before a global rollout. Following a successful release, the Edge Reboot Release (ERR) pipeline manages a systematic update and reboot of the edge infrastructure on a four-week cycle. Our control plane infrastructure typically adopts the most recent kernel, with reboots scheduled according to specific workload requirements.

By the time a CVE becomes public knowledge, the necessary fix has typically been integrated into stable Linux LTS releases for several weeks. Our established procedures ensure that we have already deployed these patches.

At the time of the "Copy Fail" disclosure, the majority of our infrastructure was running the 6.12 LTS version, while a subset of machines had begun transitioning to the newer 6.18 LTS release.

About the Copy Fail vulnerability

It helps to understand the vulnerability before getting to the response story. A comprehensive write-up can be found in the original Xint Code disclosure post.

AF_ALG and the kernel crypto API

The Linux kernel's internal crypto API manages functions like kTLS and IPsec. Userspace programs access this via the AF_ALG socket family, allowing unprivileged processes to request encryption or decryption. The algif_aead module facilitates this for Authenticated Encryption with Associated Data (AEAD) ciphers.

An unprivileged program follows these steps:

1. Opens an AF_ALG socket and binds to an AEAD template.
2. Sets a key and accepts a request socket.
3. Submits input via sendmsg() or splice().
4. Executes the operation using recvmsg().

The splice() system call is critical here, as it moves data by passing page cache references.

Memory mechanics: page cache and in-place crypto

The page cache is a shared system cache for file contents. Modifying a page belonging to a setuid binary effectively edits that program for all users until the page is evicted.

The crypto API utilizes scatterlists, which are structures linking various memory pages. In 2017, algif_aead was optimized for in-place operations, chaining destination and reference pages together. This design lacked enforcement to prevent algorithms from writing past intended boundaries.

The vulnerability: out-of-bounds write

When the user executes recvmsg(), the authencesn wrapper in the kernel performs a 4-byte write past the legitimate output region:

By using splice(), an attacker can chain a target file's page cache pages to the scatterlist. The out-of-bounds write then taints the cached file, allowing an attacker to control which file is modified, the offset, and the specific 4 bytes written. This means the attacker can manipulate the following with this exploit:

• File: Any readable file.
• Offset: Tunable via assoclen and splice parameters.
• Value: Controlled via AAD bytes 4-7 in sendmsg()

The exploit, step by step

The default exploit targets /usr/bin/su, a setuid-root binary present on essentially every distribution.

1. Cache Reference: Open /usr/bin/su as O_RDONLY and read() to populate the page cache. Use splice() on the file descriptor to pass these page cache references into the crypto scatterlist.
2. Setup: Create an AF_ALG socket, bind() to authencesn(hmac(sha256),cbc(aes)), set a key, and accept a request socket without needing privileges.
3. Write Construction: For each 4-byte shellcode chunk:
   • sendmsg() with AAD bytes 4–7 containing the shellcode.
   • splice() the binary into a pipe then the AF_ALG socket so assoclen + cryptlen targets the desired .text offset.
4. Trigger: recvmsg() initiates decryption. authencesn writes its scratch data to the target offset of /usr/bin/su in the page cache. Although the function returns -EBADMSG, the 4-byte write is now in the global page cache.
5. Execution: Running execve("/usr/bin/su") loads the tainted page cache. Since the binary is setuid-root, the injected shellcode executes with root privileges.

The upstream fix (commit a664bf3d603d) reverts the 2017 in-place optimization, removing the exploit.

How we responded 

When the vulnerability was disclosed, many workstreams started in parallel:

• Mapping the blast radius: Our security team worked with kernel engineers to determine which kernel versions were vulnerable and assess the potential exposure.
• Validating coverage: Security reviewed the exploit technique and confirmed that our existing behavioral detections could identify the exploit pattern during authorized internal validation.
• Proactive threat hunting: Security began searching for signs that the vulnerability had been exploited before it was publicly known, going back 48 hours in our fleet-wide logs.
• Engineering a mitigation: Kernel engineers began building a runtime mitigation that would protect the fleet without breaking production services.
• Continuing software updates: Our engineering teams worked on delivering an updated Linux kernel, which required carefully rebooting and rolling it out across our servers.

There was no customer impact at any point during this response.

Validating detection coverage

One of the first things our security team did was confirm that our existing endpoint detection would catch this exploit. Our servers run behavioral detection that continuously monitors process execution patterns. It doesn't rely on knowing about specific vulnerabilities; it watches for anomalous behavior across the fleet.

When our engineers validated the vulnerability internally as part of the response, the detection platform flagged it within minutes. The system linked the entire execution chain—starting at the script interpreter, moving through the kernel’s cryptographic subsystem, and ending at the privilege escalation binary—flagging it as malicious based on fleet-wide behavioral patterns.

This happened without a signature update, without a rule change, and without human intervention. Our behavioral detection coverage existed before we wrote any custom logic for this particular Copy File exploit. 

The confirmation was important because it meant we had coverage before writing a vulnerability-specific rule.

Hunting for exploitation

While our engineering team moved to a more targeted mitigation, our security investigation had been running since disclosure. This is our standard procedure for any critical vulnerability.

Our security team operates on a simple principle for critical vulnerabilities: assume compromise until you can prove otherwise. The investigation started from the assumption that exploitation could have occurred before the vulnerability was public, and we worked systematically to either confirm or rule it out.

The exploit leaves a distinctive trace in kernel logs when it runs. We searched for that trace across our centralized logging infrastructure, covering 48 hours before the vulnerability was publicly disclosed. If someone had exploited this before the world knew about it, we would have seen it.

We pulled access logs for affected systems and reconstructed who connected, when, and what commands they ran. This gave us a complete forensic picture of interactive activity on potentially affected infrastructure.

We checked that system binaries had not been tampered with, validated cryptographic hashes against known-good package manifests, looked for persistence mechanisms, and audited network connections for anything unusual. Everything was clean.

Incident timeline and impact

This graph shows the progress of our mitigation program as it progressed through our infrastructure.

How did we mitigate it?

Because of the long timeframe involved in deploying a patched Linux kernel, we also pursued mitigating this exploit without a reboot.

Removing the module

The bug was in the algif_aead kernel module. Therefore, the simple fix was to just remove this module and disallow it from being reloaded.

This mitigation was therefore exactly what the Copy Fail write-up from the security researchers who identified it recommends.

Unfortunately removing the module would have impacted software that leverages the kernel crypto API.  This meant that we had to figure out a more surgical mitigation.

Bpf-lsm

We’ve already developed and deployed such a tool for this exact scenario: bpf-lsm. Instead of removing the module, this tool leaves it loaded for legitimate users and uses a BPF Linux Security Module program to deny the socket_bind LSM hook for everyone else. This completely blocks the front door for any exploits.

A draft of the eBPF program was put together overnight. Team members picked it up the following morning, ran validations, and made it production-ready. The program is fairly straightforward. On every socket_bind call:

1. If the socket family is not AF_ALG, allow the call through unchanged.
2. If the family is AF_ALG, check the calling binary's path against an allow-list of the binaries we know to be legitimate users.
3. If the binary is on the allow-list, allow the bind. Otherwise, deny it.

To verify the mitigation on a given machine without exploiting it, the Copy Fail write-up gives a one-liner:

On a mitigated machine you get PermissionError: [Errno 1] Operation not permitted (or FileNotFoundError, depending on which mitigation is active) instead of a successful bind.

Rolling it out

Before enabling enforcement, we verified that our known internal service was the sole legitimate AF_ALG user to avoid accidental outages. We used prometheus-ebpf-exporter to hook the socket() syscall and track AF_ALG usage per binary across the fleet. This required no kernel changes and provided aggregate data from hundreds of thousands of servers within hours. Results confirmed the identified service was indeed the only legitimate user.

So the bpf-lsm rollout was deliberately staged in two steps:

1. Get visibility first. Push the ebpf-exporter config gated by salt. Confirm at the metric layer that the known service is effectively the only thing creating AF_ALG sockets.
2. Then enforce. Push the bpf-lsm program behind a separate enforcement gate.

In parallel, the upstream backport for our majority LTS line finally became available, and our internal automation built a patched kernel against it.

We started to test the patched kernel in our staging datacenters as soon as possible, then we resumed the longer reboot process in order to fully patch our fleet.

Remediation and follow-up steps

While we were prepared for this scenario, at Cloudflare we’re always learning and improving. Key areas we identified for improvement:

• Better visibility into kernel-API dependencies. We will review kernel-subsystem usage across production services, so we can continue to quickly mitigate exploits without service disruption.
• Better runtime mitigation. bpf-lsm is a valuable tool for mitigations, but we want to make this tool even better. This will include looking into faster deployments, better playbooks, and better logging and visibility of the tool. 
• Reduce attack surface of Linux Kernel. Review and audit our kernel configuration. Proactively identify unused modules or features so that we can remove them from our build entirely.

Conclusion

The "Copy Fail" vulnerability presented a unique challenge for us. Despite our practice of deploying Linux patch updates every two weeks, we remained vulnerable because a month-old mainline fix had yet to be backported to our primary kernel line. Despite that, we were still able to roll out patched kernels within hours of the backport's release. In the interim, bpf-lsm provided a surgical, no-reboot mitigation that secured our fleet. While our initial attempt to disable the problematic module failed, it did so safely within our internal staging environment rather than production, allowing us to identify this dependency.

By the end of the rollout, every machine in our fleet was protected by either a patched kernel or a bpf-lsm program denying the vulnerable code path to non-allow-listed binaries. There was no customer impact at any point during this incident, and we have committed to the follow-up work above to make our response faster and our visibility better the next time something like this lands. Responsible disclosure works, in-kernel visibility tooling pays off in moments exactly like this one, and bpf-lsm continues to be one of the most useful primitives we have for runtime kernel mitigation.

At Cloudflare, critical vulnerability response is a coordinated effort across Security, Engineering, Product, and many other teams. Special thanks to Ali Adnan, Ivan Babrou, Frederik Baetens, Curtis Bray, Piers Cornwell, Everton Didone Foscarini, Rob Dinh, Elle Dougherty, Kevin Flansburg, Matt Fleming, Kimberley Hall, Brandon Harris, Jerry Ho, Oxana Kharitonova, Marek Kroemeke, Fred Lawler, James Munson, Nafeez Nazer, Walead Parviz, Miguel Pato, Evan Pratten, Josh Seba, June Slater, Ryan Timken, Michael Wolf, Jianxin Zeng and everyone else who contributed to the investigation, mitigation, and remediation of Copy Fail. We'd also like to thank the Linux upstream maintainers and Copy Fail researchers whose work helped make a rapid response possible.