Over the past year while working on Endor Labs, I’ve spoken to dozens of CISOs. I wanted to understand their take on “software supply chain security”. It’s pretty clear at this point that this phrase became diluted almost as soon as it was coined, since it means very different things to different people. For some, the focus is on CI/CD security while for others it’s container image signing, and so on.
There was one common thread in all conversations though. A recognition of how reliant organizations are on open source software, and how little governance they had around it. For the typical modern application, over 70% of code is code you didn’t write, but depend on through open source projects. While a number of these are direct dependencies selected by developers - the actual packages required to perform a task, the majority are indirect dependencies automatically pulled into your projects. Indirect dependencies are the packages the direct dependencies depend on.
So by pulling in an unchecked amount of dependencies, you can easily lose track of the amount of packages used by your applications, and the relationships between them. Each additional dependency must be maintained, secured, and managed, and can introduce additional risk. It’s dependencies all the way down. Open source software is a huge boost to development velocity, but over the past few years we are seeing productivity stifled by security risk. And as organizations are rapidly migrating applications to cloud-native architectures (serverless, managed container services, etc.), code is all you have since your CSP abstracts and manages the rest of the stack for you.
The concept of Zero Trust is one of the most significant cybersecurity trends of the last decade. We’ve seen it applied to access management, endpoint security and of course network security. But for some reason, when it comes to open source software, we take an almost opposite approach.
Why do we treat commercial versus open source software differently?
We’ve all collectively obsessed over 3rd party risk from commercial suppliers for decades. This gave rise to whole ecosystems of risk management tools, certification standards, and assessment frameworks. In most organizations, one can’t possibly sell a commercial software or SaaS solution without completing a SOC2 attestation, conducting regular penetration testing that is independently verified, and responding to an exhaustive questionnaire. Ultimately, we’re trying to gauge the security and operational risks we’d be taking on by relying on this piece of software.
On the other hand, we seem to trust open source more implicitly. I don’t think we do this because we’re not aware of the risks. Here are a few possible reasons:
- So far we’ve followed a bit of a “don’t ask, don’t tell” policy with open source. While the gating process for commercial suppliers is managed by procurement teams, no such thing exists for open source software, where the selection process is very decentralized.
- In many organizations, we are not truly aware of the scope of direct and indirect open source dependencies. We feel that there’s a contained and acceptable risk with the hand-picked packages developers choose, and have no real visibility into the massive attack surface caused by dependency sprawl.
- It’s simply been hard to keep track. While open source is nothing new, the heavy reliance on it at the enterprise level has exploded in the last few years. And simply put, it takes a lot of time and effort to change processes at large companies.
- We confuse compliance and security. Security teams don't have many options besides scanning packages for known vulnerabilities. This works great for compliance, but doesn’t do much to actually protect applications from supply chain attacks.
The recent uptick in software supply chain attacks (650% YoY increase), high-profile incidents like Log4j, and government attention to supply chain and open source security is changing this. Attackers have recognized that this is a massive weakness in the security posture of almost every organization, and are actively exploiting it. However, we’re also seeing that attackers and vulnerabilities are only half the problem.
As highlighted by Log4j, the operational overhead of the response to a vulnerability is almost as dangerous as the potential breach itself. During this incident, organizations had to figure out where they were using Log4j, and even who was using anything that uses Log4j. And while the Apache Software Foundation reacted swiftly and issued an update, even the decision to patch the package across thousands of repositories is risky. Organizations have to try and analyze the potential impact of updating versus living with the vulnerability.
The result is that hundreds, if not thousands of organizations still have the Log4j vulnerability in their systems. And while there aren’t too many reports of large scale exploitations of the vulnerability, the real negative impact became clear during the response. Even the most advanced engineering organizations spent thousands of hours digging through complex dependency graphs to find where the problems actually were. The recent CSRB report on the incident mentions that “one federal cabinet department reported dedicating 33,000 hours to Log4j vulnerability response to protect the department’s own networks. These costs, often sustained over many weeks and months, delayed other mission-critical work, including the response to other vulnerabilities.”
So, how do we move forward?
Zero Trust Dependency Management
The philosophy behind Zero Trust security is pretty simple - “never trust, always verify”. According to Palo Alto Networks: “Zero Trust is a strategic approach to cybersecurity that secures an organization by eliminating implicit trust and continuously validating every stage of a digital interaction. Rooted in the principle of “never trust, always verify,” Zero Trust is designed to protect modern environments and enable digital transformation by using strong authentication methods, leveraging network segmentation, preventing lateral movement, providing Layer 7 threat prevention, and simplifying granular, “least access” policies.”
Hundreds of millions of dollars are spent every year applying these principles to IAM, network, and endpoint security. Yet, often not intentionally, organizations currently consuming open source software do so with a “trust anyone” mentality. Why put guards and theft deterrent systems on one door, only to leave the other one wide open?
The time has come to move away from implicit trust, to Zero Trust dependency management. Organizations can leverage the immense power of open source, and continue to contribute and strengthen the community, without exposing themselves to supply chain risks - and the whole ecosystem will be better for it.
Here’s an outline of requirements for Zero Trust dependency management:
- Know what you rely on - Have complete visibility and tracking for every package in your codebase. Not just the direct dependencies that developers intentionally download and use, but also the tens of thousands of indirect dependencies those packages automatically bring in. This is the only way to maintain a complete Software Bill of Materials (SBOM).
- Assist with dependency selection - Developers use open source dependencies to build faster, and to focus on building the company’s unique products. To help them do so, and not create the traditional developer/security friction, instant feedback on dependency selection is required. This feedback needs to be based on both operational and security risk.
- Measure multidimensional risk - Risk must be assessed across multiple dimensions including operational risk, security risk, and attack patterns. It’s not enough to rely on known vulnerabilities, which typically are accidental mistakes made by well intentioned developers, and miss a whole class of supply chain attacks.
- Separate compliance from security - Tasking developers with an endless stream of vulnerabilities to fix erodes trust, wastes time, and often hurts the security posture of the supply chain since more critical issues are not addressed. Security leaders need the outputs from security tools to be actional and believable to earn your developers’ trust, and to make sure remediation efforts are quick.
- Prioritize maintenance and patching - Continuously updating dependencies in large environments often leads to unexpected breaking changes and risk for those who depend on them. These risks can be contained by analyzing the potential impact of an update, and the prioritization of dependencies that are actually reachable and exploitable.
The open source community will continue to grow and build amazing things. To both accelerate this thriving ecosystem and make sure engineering organizations can continue to rely on it, we need to change our approach. The time to make these changes is now. We’re at a turning point that exists in the evolution of every technology trend - either we evolve our security practices with it, or we risk creating an environment where open source is not trusted in the enterprise, and innovation is stifled.