.png)
.png)
.png)
.png)
.png)
.png)
.png)
.png)
.png)
.png)
.png)
.png)
.png)
XZ is A Wake Up Call For Software Security: Here's Why
The xz backdoor shines a light on everything we're doing wrong in software supply chain security. Get an overview of the incident, what we can learn from it, and what we can do about it.
The xz backdoor shines a light on everything we're doing wrong in software supply chain security. Get an overview of the incident, what we can learn from it, and what we can do about it.
The xz backdoor shines a light on everything we're doing wrong in software supply chain security. Get an overview of the incident, what we can learn from it, and what we can do about it.
The xz backdoor shines a light on everything we're doing wrong in software supply chain security. Get an overview of the incident, what we can learn from it, and what we can do about it.
The xz backdoor shines a light on everything we're doing wrong in software supply chain security. Get an overview of the incident, what we can learn from it, and what we can do about it.
“Reports that say that something hasn't happened are always interesting to me, because as we know, there are known knowns; there are things we know we know. We also know there are known unknowns; that is to say we know there are some things we do not know. But there are also unknown unknowns—the ones we don't know we don't know. And if one looks throughout the history of our country and other free countries, it is the latter category that tends to be the difficult ones”. "Defense.gov News Transcript: DoD News Briefing – Secretary Rumsfeld and Gen. Myers". United States Department of Defense. February 12, 2002”
The unexpected can sometimes teach us the most. Donald Rumsfeld's remarks on February 12, 2002, capture a crucial insight into the nature of risks and uncertainties. He distinguished between what we know, what we know we don't know, and what we don't realize we don't know—the "unknown unknowns." It's this last category that often presents the greatest challenges, especially in cybersecurity, as recent events have starkly illustrated.
The XZ Incident is a Wake up Call
The cybersecurity landscape was recently rocked by what could have narrowly become the most significant threat of its kind. The XZ incident exposed an attack within an open-source package, which is an essential building block of the Linux ecosystem. A sophisticated adversary masqueraded as a legitimate contributor to a widely-used open-source project, embedding a backdoor over two years. This backdoor had the potential to compromise millions of Linux systems globally and was only discovered by accident. The plot reads like the premise for a gripping techno-thriller, but it underscores a frightening reality: our critical technological infrastructure is under constant threat from well-organized and funded entities, possibly state-sponsored or linked to organized crime.
This incident has underscored the vulnerability of open-source projects to insider threats, a risk previously confined to the corridors of critical technology companies. These companies have long employed rigorous security measures such as multiple code reviews and hermetic pipelines to guard against such risks. However, the XZ attack reveals that the open-source ecosystem is equally at risk, suggesting the presence of other, undetected vulnerabilities lying in wait.
The State of Software Supply Chain Security
The XZ attack has laid bare the inadequacies of the current approach to software supply chain security. Analogous to bringing knives to a gunfight, the software industry finds itself ill-prepared to counter such sophisticated threats. Vulnerabilities known as CVEs represent the "known knowns," yet the industry struggles even to keep pace with these. The reliance on open-source software, often perceived as "maintenance-free," exacerbates the problem. Meanwhile, oversight organizations like NIST and NVD are overwhelmed, further compounding the issue.
Ineffective Strategies
Blaming maintainers or restricting anonymous contributions are not viable solutions. The open-source community thrives on anonymity and voluntary contributions, and attackers can easily circumvent measures aimed at limiting anonymity through sophisticated social engineering techniques. States and organized crime can use any of the techniques in spy history to force their way.
Avoiding open source software is also not an answer. We were lucky that the attack happened against open source software that anyone can look at and understand. If the same attack was against a closed source component, how would we even know?
What Can We Do About the Next XZ?
Other than going off-the-grid and isolating in a cabin in the wilderness, can we do anything else to protect ourselves?
Acknowledging the reality of cyber warfare against our infrastructure is the first step toward defense. No single entity can tackle this challenge alone, but early detection and vigilance can help mitigate risks.
Vet Auxiliary Dependencies
Let’s start with the now famous Linus’s Trovals' law: “given enough eyeballs all bugs are shallow”. The “keyword” in the above statement is “enough”.
In the software ecosystem, you can think of two types of dependencies. Foundational open source software (OpenSSL, Linux, Spring Framework in Java, Kubernetes, etc). These dependencies are supported by whole communities, they have contribution guidelines and are probably some of the “safest” since there are “enough” eyeballs.
Auxiliary dependencies are “small” one-off-packages, often maintained by one or two individuals. These dependencies can often be replaced and using them is a “choice” that any software organization has. Rob Pike coined the statement: “a little copying is better than a little dependency”. There are a set of rules that one has to follow for these dependencies before allowing them in their code-base, such as:
- Do they have proper security controls such as code reviews or does a single individual commit all code?
- Do they have binary files in their repositories that nobody knows what they are?
- Do they follow any practices of automated build/releases processes?
- Do they have more than one maintainer?
- Are they actively maintained?
For these dependencies an organization has to assume that “once they are used” they are “owned” by the organization. They have to be treated like your “own” code. They have to be understood in detail by your own engineers and very often you are better off just copying them.
The OWASP Top 10 Risks for Open Source Software (OSS-Risk-2) identify these patterns as a significant threat. There are open source efforts like the OpenSSF scorecard and commercial products that have started work around addressing these warning signals.
The Curse of Transitive Dependencies
Even with due diligence on direct dependencies, transitive dependencies pose a hidden danger. Assuming that you have done your job as above and vetted your auxiliary direct dependencies, the problem is still not solved. The systemic issue requires a collective response.
If the maintainers of “foundational” dependencies bring in transitive auxiliary dependencies that do not meet the requirements above, we end up in the same situation. In the XZ attack, few people import this package directly. It is because of “systemd”, a foundational dependency that the attack is activated.
And this is exactly where proper funding and engineering mechanisms have to be applied. Very often these foundational open source projects are funded by big corporations that fail to recognize the importance of proper vetting of their dependencies. Maybe it is time to call out the practices of some of these entities and demand accountability.
Conclusion
The XZ attack is a stark reminder of the evolving landscape of cyber threats, shifting from the “lone hacker” to “state sponsored attacks” and from potential "unknown unknowns" to "known unknowns." As we navigate this challenging terrain, a concerted effort across the industry is required. By reevaluating the management of open-source dependencies and embracing collective responsibility, we can strengthen our defenses against the unseen dangers that lurk in the digital shadows.