Defending against supply chain attacks like Chalk/Debug and the Shai-Hulud worm

Building on top of open source packages can help accelerate development. By using common libraries and modules from npm, PyPI, Maven Central, NuGet, and others, teams can focus on writing code that is unique to their situation. These open source package registries host millions of packages that are integrated into thousands of programs daily.

Unfortunately, these key services are prime targets for threat actors looking to distribute their code at scale. If they can compromise a package in one of these services, that one action can automatically affect thousands of other systems.

It started with compromised credentials for a trusted maintainer for npm. After social engineering the credentials, 18 popular packages (including Chalk, Debug, ansi-styles, supports-color, and more) were updated with an injected payload.

This payload was designed to silently intercept cryptocurrency activity and manipulate transactions to the bad actor’s benefit.

Together these packages are downloaded an estimated two billion times each week. That means even with the rapid response from the maintainer and npm, the couple of hours that the compromised versions were available could have led to significant exposures. Any build systems that downloaded the packages during this window or sites that loaded them remotely were potentially vulnerable.

This sophisticated malware used intelligent reconnaissance techniques and adapted its behavior to find the most effective attack vector for its current context.

The very next week, the Shai-Hulud worm started to spread autonomously through the npm trust chain. This malware uses its initial foothold in a developer’s environment to harvest a variety of credentials, such as npm tokens, GitHub personal access tokens, and cloud credentials.

When possible, the malware would expose the harvested credentials publicly. When npm tokens are available, it publishes updated packages that now contain the worm as an additional payload. The now compromised packages will execute the worm as a postinstall script to continue propagating the infection.

In addition to this self-propagation method, the worm also attempts to manipulate GitHub repositories it gains access to. Shai-Hulud sets up malicious workflows that run on every repository activity, creating a resilient and continuous exfiltration of code.

This exploit showed technical sophistication and a deep understanding of the developer workflows and the trust relationships that power the community. By using the standard npm installation processes, the worm makes detection more challenging because it operates within the behavioral patterns expected of developers.

Within the first 24 hours of this exploit, over 180 npm packages had been compromised, again potentially affecting millions of systems. Both incidents show the potential scale of supply chain compromises.

If a compromised package has made it into production, you should follow your standard incident response process for active incidents to resolve the issue.
To sweep your development environment, we recommend the following steps:

1. Audit dependencies: Remove or upgrade to clean versions of Chalk and Debug packages and check for Shai-Hulud-infected packages.
2. Rotate secrets: Assume npm tokens, GitHub PATs, and API keys might be compromised. Rotate and reissue credentials immediately.
3. Audit build pipelines: Check for unauthorized GitHub Actions workflows or unexpected script insertions.
4. Use Amazon Inspector: Review Amazon Inspector findings for exposure to the Chalk/Debug exploit or Shai-Hulud worm and follow recommended remediation.
5. Harden supply chains: Enforce SBOMs, pin package versions, adopt scoped tokens, and isolate continuous integration and delivery (CI/CD) environments.

We regularly share the findings from the malicious package detection system in Amazon Inspector with the community through our partnership with the Open Source Security Foundation (OpenSSF). Amazon Inspector uses an automated process to share this type of threat intelligence using the Open Source Vulnerability (OSV) format.

Amazon Inspector employs a multi-layered detection approach that combines complementary analysis techniques to identify malicious packages. This approach provides robust protection against both known attack patterns and novel threats.

Starting with static analysis using an extensive library of YARA rules, Amazon Inspector can identify suspicious code patterns, obfuscation techniques, and known malicious signatures within package contents. Building on that, the system uses dynamic analysis and behavioral monitoring to identify threats, despite their use of evasion techniques. The final set of analysis is conducted using AI and machine learning models to analyze code semantics and determine the intended purpose versus suspicious functionality within packages.

This multi-stage approach enables Amazon Inspector to maintain high detection accuracy while minimizing false positives, helping to make sure that legitimate packages are not incorrectly flagged and sophisticated threats are reliably identified and mitigated.

When these threats are detected in open source packages, the system starts the automated workflows to share this threat intelligence with the OpenSSF. This workflow sends the validated threat intelligence to the OpenSSF where the contributions are rigorously reviewed by the OpenSSF maintainers before being merged in the community database. That is where they receive an official MAL-ID or malicious package identifier.

This process helps verify and share these types of discoveries as quickly as possible with the community, so that other security tools and researchers benefit from the detection capabilities of Amazon Inspector.

Chalk/Debug and the Shai-Hulud worm are not novel exploits. These are—unfortunately—the most recent incidents using this vector. Open source repositories are a fantastic resource for developers and help many teams to innovate more quickly. The open source community is working hard to reduce the impact of these types of incidents.

That is why we have partnered with the OpenSSF and have contributed reports that highlight over 40,000 npm packages that were compromised or created with malicious intent. We believe that Amazon Inspector is an excellent tool to help you build safely and securely, and while we would love everyone to use it, we are proud that our work and contributions to efforts like OpenSSF are helping improve the security of everyone in the community.

---

If you have feedback about this post, submit comments in the Comments section below. If you have questions about this post, add a note in AWS re:Post tagged with Amazon Inspector, or contact AWS Support.