Software Bill of Materials (SBoM) – Does It Work for DevSecOps? ~ Hacking Techniques

There has been much discussion of a “software bill of materials” (SBoM) lately, for use when addressing security vulnerabilities. Many are curious, wanting to learn more. Googling the term gives lots of positive descriptions. This post will go negative, describing problems with the concept.

Rather than cover the entire concept, I want focus on a narrow part of it, so I asked Kate Brew to write a short blurb why she’s interested in SBoMs. Her response was:

“I am an Industrial Engineer by training. So when I heard of the concept of software BoM I was intrigued. Being able to quickly see all the components, open source or not, incorporated into an application appears like a valuable way to determine needed actions in the case of vulnerabilities found in a component. It seems efficient and helpful to me to have a clear view of components in an application.”

Software is never built wholly from scratch these days. Instead, software is built combining components, development frameworks, libraries, operating system features, and so on. It has a “bill of materials” describing the bits that make it up every much as hardware does.

When vulnerabilities happen, knowing this information can help. Good examples are the high profile Apache Struts bugs, where customers don’t know they are vulnerable because they are unaware that products they own include Struts. If only product vendors provided a list of sub-components, then customers would quickly know if they are vulnerable, and be able to act accordingly.

Some claim this sort of thing already exists in narrow industries, like medical and energy. They are pushing the concept for use everywhere because it’s already being used successfully somewhere.

This is a great story, but it isn’t true.

Software Bill of Materials Is a Misguided Concept for DevSecOps

Proponents are being deliberately vague defining exactly what should be in included in a software BoM. For hardware BoMs, you don’t list the ingredients of the circuit board, where you sourced the silica for glass fibers, or the recipe of the epoxy that binds them together. Hardware BoMs aren’t that granular because it’s not necessary. They include an indented list of components and sub-components. Hardware is basic. But when tracking software vulnerabilities, such granularity is important: you need to track every line of source code.

There are four levels of details for SBoMs:

Licenses
Modules
Patch levels
Backports

Most of the discussion about SBoMs is roughly at the license level. The makers of software already track this, even when they don’t disclose it to customers. Commercial products track this for legal reasons, for compliance with legal contracts they have with suppliers. Open-source products track this for practical reasons, since you often have to hunt down install the dependencies yourself in order to make open-source work — importing open-source implicitly means importing the license.

You see the artifacts of this everywhere. My parents just bought a new Subaru, which like most new cars contains a small screen for the maps and backup camera. On one of the pages on the screen I find something that lists a number of embedded components. Displaying this information is often a requirement of the license.

Software Bills of Materials Aren’t That Great for Tracking Vulnerabilities

SBoMs aren’t as useful as you’d think for tracking vulnerabilities, because it’s not granular enough. Take Linux, for example. The entire thing is licensed under the GPL. This hides the complexity that the kernel is around 20 million lines of code, and the GNU userland components are millions more. An SBoM saying this IoT product uses “Linux” hides a lot of the complexity of what may or may not exist in the product.

A new Linux vuln is discovered at the rate of about once per week. Using the above example of tracking Struts vulns, it means you’d be diligently patching most of the devices in your organization once per week. The paragraph above says “if a vulnerability” is found. The reality is “when”, and they are found constantly. That Struts example is cherry picked to show how the SBoM-vuln process could work sometimes, but is not representative of how this process would work most of the time. The way it would work most of the time is that you’d have a team of people checking vulns every morning and pushing out the patches.

Lists of Component Licenses Doesn’t Work in Software BoMs

We don’t want an SBoM built from just the existing list of licenses, but something listing the subcomponents. When a subcomponent of Linux has a bug, we only want to patch the stuff using that subcomponent. For example, many IoT devices use BusyBox and uClibc instead of the GNU glibc userland code. When a glibc bug is found, you don’t to patch all the devices using uClibc instead.

While there are many examples of SBoMs at the software license level, I’ve found none that successfully track things at the module/sub-component level. A good measure of this is looking at popular modules like zlib and pcre. These are used everywhere. Most large software product includes them, even Windows itself. However, they have easy licensing terms, and are buried deep in the SBoM hierarchy, as a component of a component of a component, rather than an easily identifiable piece itself, so they are rarely seen. If you pretend there is a vulnerbiltiy in zlib and track it down, you’ll find it in less than 10% of your products, when in fact, something like 80% of your products depend upon it. When people claim they are already successfully using SBoMs to track vulnerabilities, I’m not sure they really are, since I know they aren’t tracking subcomponents with sufficient granularity.

Listing all the licensed modules gets you around 10 items per product, but trying to include all the subcomponents can easily balloon to thousands. Modern software is complex. The average, simple app for your phone is often the work of only a couple developers, but is megabytes in size, because of the enormous system of sub-components that make up the software.

Figure – First level dependencies of the Apache web server

IoT, Software Bills of Materials and Other DevSecOps Complexities

And this is still only scratching the surface. By the time you get software, it’s already out of date, especially for commercial products and IoT devices, which can be years out of date. You aren’t listing just the subcomponents, but also their versions.

This includes patch levels. When version 4.6 of the outdated module has a vuln, the vendor doesn’t upgrade to the latest version at 5.2 where the vuln is fixed, because that’ll introduce bugs into their product. Instead, they’ll upgrade to version 4.6.1, with just the added patch. Thus, checking whether you are vulnerable doesn’t mean checking if you have the latest version of the software, like 5.2, but whether you have the latest of the old version of the software like 4.6.1.

Popular libraries like zlib are a good example. Not only do you have it everywhere throughout your enterprise, you have every version and patch level someplace as well, going back decades. When proponents claim that SBoMs exist and are being used, they don’t mean at this level of granularity.

But there is still more. When a vulnerability is found, the owner of a module doesn’t go back and patch every known version. They only patch a few of the most recent versions and tell everyone else to upgrade. In recent years, “LTS” (long term support) versions of modules have become popular, where the owners of modules declare the subset of versions that they will continue to support, and for how long.

In the open-source world, if the owner of a module doesn’t provide a patch for your version of the module, then you’ll have to handle it yourself. This is known as a “backport”. Usually the fix applied for a later version of the module will work cleanly for earlier versions as well, so backports are often an easy process.

While there are efforts to track subcomponents, versions, and patch levels, nobody has even attempted to track backports. The problem is pretty much insurmountable. When the owner/maintainer of a project updates the software, that’s a single event that can be named and tracked. When many different people apply backports to an older version, these are all separate events, often all slightly different.

Software, especially in IoT devices, has lifespans of decades. Even LTS “long term support” version rarely continue supporting software for more than 5 years. Thus, most products spend most of their lifecycle in this area where backports are a thing. When people talk about SBoMs tracking vulnerabilities, they are talking about only a small subset of products, tracking the latest versions. You can force vendors of the products to continue to support them for decades, but those vendors can’t force their suppliers of subcomponents to continue supporting them, especially open-source components.

Practical DevSecOps Considerations for Software Bills of Materials

But even if a magic faerie could fix these problems, then you’d simply have an overload of work, because this would tell you about a lot of vulnerabilities. Just because a module has a vulnerability doesn’t mean the entire package using the module is vulnerable. Not all users of the latest Apache Struts is vulnerable to their latest vulnerability, for example, but only those that used a specific feature of Struts.

Weld Pond, CEO of VeraCode, has looked into this. They have a static analyzer for binary code that can often determine whether a software package can reach the vulnerable code in a subcomponent. They find that most of the time, a vulnerability in a subcomponent isn’t practically exploitable. In other words, a vulnerability in a module means that some users are vulnerable, but not automatically all of them, or even most.

Using SBoMs to track vulnerabilities gives you two unappealing options. You can either try to patch every time there is a match, or do your own analysis to see if you are actually vulnerable and whether the patch is needed. Either way is expensive. Or, you can rely upon the vendor of to make this determination, and only apply the patches the vendor recommends. But that’s the situation were are in today without SBoMs.

Example: Heartbleed. Would Software BoM Help?

The OpenSSL Heartbleed bug form 2014 is a good example. Some of your stuff is still vulnerable to this. How will an SBoM help you out?

Using SBoMs at the software license level won’t help you much. Since OpenSSL is often a component of a component, it’ll get missed, so the list of products using OpenSSL will be partial. Since the bug is only server side, whereas OpenSSL is most often used in clients, most of that list that you do get will be of packages that aren’t actually vulnerable to Heartbleed. In other words, you’ll simultaneously miss needed patches while applying unneeded ones.

Katie Moussouris, formerly the manager of Microsoft’s vulnerability disclosure program, describes how Microsoft, which internally tracks software modules with things like SBoMs, still took months to figure out exactly which of their products and services were vulnerable to Heartbleed. It’s not the “easy” solution people imagine.

This is why I’m dubious of claims that SBoMs already exist and are being used successfully in some places. This cannot possibly be the case.

I feel the situation is similar to the TSA, the airport passenger screeners. They regularly post (https://www.tsa.gov/blog/tags/year-review) about the thousands of firearms they block from getting on airplanes. The implication is that TSA screening works.

But it doesn’t. Independent audits going back years show that 80% of firearms get through (https://abcnews.go.com/US/tsa-fails-tests-latest-undercover-operation-us-airports/story?id=51022188). The system is deeply flawed, and their measurement of success comes from cherry picking the type of measurement they use, such as “guns blocked” rather than “guns let through”.

Conclusion: Do Software Bills of Materials Really Work in Real Life for DevSecOps Practitioners?

That’s how SBoM-vuln programs are “successful”. They are in fact, missing most vulnerabilities, but it’s not a problem because other mitigations (like firewalls) block most of the attacks anyway. At the same time, a lot of the patches they do apply are unnecessary, but nobody notices when you overspend on useless security, because everyone agrees it’s your moral duty to spend a lot on security. There is so far no example of a successful SBoM program that hasn’t fudged its numbers like the TSA.

Forcing companies to publish the SBoM at the license level isn’t necessarily a bad thing, since they already track the information. This will weed out those companies that have such crappy development processes that they can’t track the information. This will also weed out ethically challenged companies like Oracle that hide this information because they are in violation of software licenses (like the GPL).

But it won’t tell you all the information you need for vulnerability management. SBoMs today are not nearly granular enough, and simply matching vulnerabilities still doesn’t tell you whether you are actually vulnerable.