Lessons from SolarWinds – Evaluating third-party libraries

In December 2020, SolarWinds disclosed that their Orion product had been subject to a large-scale attack which in turn granted the attacker a backdoor into the infrastructure of many SolarWinds clients, notably including multiple government agencies. The scope of this breach has brought a great deal of attention to supply chain vulnerabilities and the role of third-party dependencies in the security of a software system. While none of these threats are new, the additional scrutiny brought to this area by current events has left organizations with a strong desire to improve their security posture in hopes of not being the next target. As a software engineering team, your most common interaction with the supply chain is in the choice of external libraries to incorporate into your applications.

A well-maintained external library can be a huge boon to development efforts, saving countless hours of engineer time by allowing you to focus on your own product’s core differentiators rather than needing to reimplement common functionality. Conversely, poorly implemented libraries can be difficult to use, introduce security vulnerabilities that create risks for your organization and your customers, or in extreme cases even interfere with the workings of unrelated parts of your service or undermine your entire architecture. Here I’ll outline many of the considerations I’ve used to evaluate libraries for the soundness of their engineering practices and their likely security implications. I offer these approaches as a starting point, not a prescription. Find an approach that meets your own needs. While there are no silver bullets in this space, ensuring that your project relies only on high-quality dependencies can substantially reduce your attack surface and make your product easier to expand and maintain.

Library scope

This initial consideration is key because it determines how thorough an evaluation is necessary or appropriate. How much functionality does the library provide? How important is this functionality to the functioning of your product? A simple library with a small footprint requires far less scrutiny than a large framework that will be tightly integrated into your own code. And some indicators that would be major red flags for key functionality (e.g., infrequent updates) may be normal or expected for smaller libraries that are only trying to accomplish an extremely narrow task.

A related question is how you intend to use the library. Development dependencies (for example, a static analysis tool or a component of a unit test framework) present far less risk than runtime dependencies. While failures in your build pipeline can delay a release or cause other short-term problems for developers, they aren’t directly user-impacting and typically aren’t a risk to the security of the application itself.

Maintenance history

The quickest way to get a sense of a project’s overall health is to look at its source control history. When was the last release? How frequently are releases published, and what is the rate of commits to the source code repository? I like to see projects that are updated frequently, because this gives me greater confidence that the library will continue to be maintained into the future. Conversely, a project that has sat dormant for a long time may contain unaddressed bugs or might introduce compatibility issues with new versions of common dependencies or even the language runtime itself.

Looking through a library’s changelog is also extremely valuable. Sometimes the changes introduced by each release are described on the releases page. Other times the changelog is a file in the top level of the source code repository. If this information isn’t readily available, that’s a major red flag. I look for clear descriptions of what changed from one release to the next and that any backwards incompatibilities are clearly documented. However, a high frequency of backwards-incompatible changes or bug fixes to core functionality is itself cause for concern as this suggests that the developer does a poor job of planning and debugging changes prior to release.

Maintainer relations

A more in-depth evaluation might involve looking through a project’s issue tracker and other public communications channels. How responsive are the maintainers to users? Are they friendly and receptive to feedback, or do they often take a more combative tone? What is the ratio of opened to closed issues? Are maintainers actively addressing user concerns, or are bug reports lying fallow for long periods of time? How extensive is the documentation, and how effectively does it point the user towards implementation best practices? Is there evidence of a vibrant community around this project? For example, does the burden of user support fall entirely on the primary maintainer, or are there other community members willing and able to pick up some of the slack? A project that maintains a healthy relationship with its community is much more likely to meet users’ specific needs and adapt quickly to a changing environment.

Transitive dependencies

This evaluation takes a bit more effort than simply counting the frequency of releases, but provides a treasure trove of information. The first thing I want to understand is how many additional dependencies a library relies on, particularly whether it pulls in dependencies needlessly. Dependencies that provide key functionality for a library are much less concerning than pulling in many additional libraries for unrelated tasks. Each additional dependency means added complexity and additional supply chain risk. If in doubt, one simple evaluation tool is to install the library in a docker container and note how many additional libraries are added in the process.

The biggest concern in this category is how easily we can update dependencies as new vulnerabilities are discovered. For this reason, an important piece of the evaluation is noting whether a dependency is restrictive (only satisfiable by a specific version of a library) or permissive (satisfiable by a range of versions). Restrictive dependencies are a major concern because they can make patching efforts difficult or even force downgrades of libraries we depend on for other functionality. Also relevant is how current these transitive dependencies are. A versioned dependency specifying a two-month-old release is much less concerning than a dependency on a two-year-old release. In a similar vein, one sign of a well-run project is a history of frequent updates that keep its dependencies fresh.

Cross-language dependencies

Be especially wary of packages written in one language with additional dependencies in a different language (for example, Python libraries with compiled-in C extensions). If the secondary dependencies involve a language or programming environment that you’re less familiar with, it will be harder to evaluate their overall quality. Furthermore, different languages often have different packaging conventions, and maintainers of mixed-language projects are often not well-versed in the best practices for all the programming languages they operate in, making patching and updating even more difficult.

Cryptographic libraries

Cryptographic libraries merit special consideration due to the algorithmic complexity involved and the high impact of potential vulnerabilities in this space. Vet these libraries especially carefully and ensure that you’re relying only on thoroughly reviewed libraries with a long track record. Some safe choices are OpenSSL for C and C++, BouncyCastle for Java, and PyCA’s cryptography library for Python. Your options may be restricted by your organization’s security policy or compliance requirements. Be aware of known vulnerabilities in cryptographic libraries and stay informed of new releases. Of all your third-party dependencies, these have the greatest security impact.

Code and documentation quality

This section and the next are high-effort evaluations, but this level of scrutiny is sometimes justified in higher-risk cases. When I want to get a sense of a project’s overall code quality, I try to find source code that uses functionality I already understand deeply and evaluate how well they’ve written that small area of their codebase. Sometimes I’ll look over the quality of the project’s documentation, focusing on the writer’s clarity and ability to effectively communicate complex concepts. In either case, the goal is to determine whether the maintainers show high attention to detail or instead tend to cut corners. The expectation here is that the degree of care shown to one part of the project is likely to be representative of the state of the whole.

As a concrete example, if I’ve chosen to look at a Python project’s handling of SSL, I might notice the following:

Do they use default SSL connection parameters or try to customize them to more secure settings?
If they make customizations, how sensible are the changes?
Do they allow the user to make additional customizations?
If they allow customizations, do they reuse Python’s standard library SSLContext objects or try to reinvent the wheel?

In an evaluation like this, I want to see sound design principles, reuse of standard idioms, and that the author can foresee potential user needs and provide appropriate extensibility to address them.

Vulnerability handling

I’ve listed this last because it’s usually an impression I build up over time and can be less useful for an initial evaluation. It’s also heavily dependent on how widely used and security-relevant a particular library is. However, this can certainly be a factor in deciding whether to continue to use an existing library or attempt to migrate to a replacement.

Simply put, I try to evaluate the project maintainer’s responsiveness in patching vulnerabilities and how easy it is to gain the benefits of those patches in our own deployments. A project that’s frequently subject to vulnerability disclosures is somewhat concerning, but could just as easily be a factor of wide use and security relevance rather than overall code quality and maintenance practices. A better metric is what happens after a vulnerability becomes known. Are patches incorporated into new releases quickly? How easy is it to upgrade once patched versions are released? If the project often makes sweeping changes, particularly if they regularly break backwards compatibility, do they backport fixes to old versions? If the library is part of a Linux distribution, then responsibility for backporting patches usually falls to the distro maintainers rather than the original developer, but the same questions can be applied to how the packager handles the library (distributions often make a distinction between core packages with active maintenance and more peripheral packages, which typically track upstream releases with minimal changes applied).

When a library shows a clear history of handling vulnerabilities poorly or operates in a way that makes it difficult to take advantage of security patches, I’ll often recommend that my team migrate to a replacement.

Concluding thoughts

Evaluating the overall quality of a software project and its surrounding community is an essential skill that will only get more relevant as software projects become more interdependent. It’s also a highly individual skill; you will often get the most mileage from investigating the decisions library maintainers make within your areas of expertise. Because fully validating every detail of a potential supplier is an even more difficult task than building the functionality yourself, the evaluation process becomes one of establishing a degree of trust and taking steps to ensure that trust isn’t misplaced. Ultimately, learning to vet third-party dependencies is an important component of building an engineering culture that takes ownership of your product’s security and reliability.

About the author

Jacob Emmert-Aronson is a senior engineer on the Mindmeld team, part of the Webex Intelligence organization. He is a knowledge leader in information security, DevOps, and software maintenance, and his specialty is understanding how these topics intersect