A common question came up again this week working with a developer (and friend) at a partner that does custom software development:
I’m working on a project that is implementing a bunch of Node microservices. What we are realizing is that we aren’t sure of the best way to manage package dependencies across multiple micro-services managed as different projects in bitbucket. So, we’re balancing trying to be reasonably current with our included packages to avoid security (and other) issues against the insanity of doing nothing but constantly updating packages across multiple projects, updating references to maintain consistency (as well as minimize the number of potential packages/versions that could have vulnerabilities), etc.
There really are a bunch of competing factors when we think about managing dependencies. Security folks generally tell developers to update immediately and developers resist with tones similar to the one in the quote above about “insanity of doing nothing but constantly updating packages”.
This post offers our most common advice on the topic.
It is Hard
First and foremost, let me step back and acknowledge that this is harder than it looks at first glance.
When we train on this, the developers in the room offer lots of reasons that it is challenging. Why is it hard? Well, in a java stack based application I might see 50-100 dependencies (without really trying). In Node, because of the proliferation of small packages, we often see thousands of dependencies. It is actually unsurprising in this context that there would be changes to a package every day.
Some changes are important, others are not. Just because a library has a serious vulnerability in it doesn’t mean that the code that uses the library exposes that vulnerability. If I’m using a database library and there is a vulnerability in
methodXYZ() but I never call
methodXYZ() then is my code vulnerable?
Not only that, often library updates introduce breaking changes that require code to be rewritten. I was bitten when Hibernate updated the default database key column type from using Integers to using Longs. I never thought to write unit tests that checked the types and we rolled an update to try to incorporate Hibernate updates and the whole system blew up in our face. We needed to have done more testing and/or manual QA to ensure that nothing under the hood of that update was going to break existing functionality. Similar things happen with libraries all the time. But many organizations can’t do more testing very easily.
Here’s an example of what we get when I run the current
npm audit on an older version of our JASP API.
om:jasp-api mk$ npm audit === npm audit security report === # Run npm install firstname.lastname@example.org to resolve 3 vulnerabilities SEMVER WARNING: Recommended action is a potentially breaking change ┌───────────────┬──────────────────────────────────────────────────────────────┐ │ Moderate │ Prototype Pollution │ ├───────────────┼──────────────────────────────────────────────────────────────┤ │ Package │ lodash │ ├───────────────┼──────────────────────────────────────────────────────────────┤ │ Dependency of │ eslint │ ├───────────────┼──────────────────────────────────────────────────────────────┤ │ Path │ eslint > inquirer > lodash │ ├───────────────┼──────────────────────────────────────────────────────────────┤ │ More info │ https://nodesecurity.io/advisories/782 │ └───────────────┴──────────────────────────────────────────────────────────────┘ ... # Run npm install email@example.com to resolve 2 vulnerabilities ┌───────────────┬──────────────────────────────────────────────────────────────┐ │ Moderate │ Prototype Pollution │ ├───────────────┼──────────────────────────────────────────────────────────────┤ │ Package │ lodash │ ├───────────────┼──────────────────────────────────────────────────────────────┤ │ Dependency of │ aws-sdk │ ├───────────────┼──────────────────────────────────────────────────────────────┤ │ Path │ aws-sdk > xml2js > xmlbuilder > lodash │ ├───────────────┼──────────────────────────────────────────────────────────────┤ │ More info │ https://nodesecurity.io/advisories/782 │ └───────────────┴──────────────────────────────────────────────────────────────┘ ... # Run npm install firstname.lastname@example.org to resolve 1 vulnerability ┌───────────────┬──────────────────────────────────────────────────────────────┐ │ Moderate │ Code Injection │ ├───────────────┼──────────────────────────────────────────────────────────────┤ │ Package │ morgan │ ├───────────────┼──────────────────────────────────────────────────────────────┤ │ Dependency of │ morgan │ ├───────────────┼──────────────────────────────────────────────────────────────┤ │ Path │ morgan │ ├───────────────┼──────────────────────────────────────────────────────────────┤ │ More info │ https://nodesecurity.io/advisories/736 │ └───────────────┴──────────────────────────────────────────────────────────────┘ # Run npm update cryptiles --depth 5 to resolve 2 vulnerabilities ┌───────────────┬──────────────────────────────────────────────────────────────┐ │ High │ Insufficient Entropy │ ├───────────────┼──────────────────────────────────────────────────────────────┤ │ Package │ cryptiles │ ├───────────────┼──────────────────────────────────────────────────────────────┤ │ Dependency of │ @sendgrid/mail │ ├───────────────┼──────────────────────────────────────────────────────────────┤ │ Path │ @sendgrid/mail > @sendgrid/client > request > hawk > │ │ │ cryptiles │ ├───────────────┼──────────────────────────────────────────────────────────────┤ │ More info │ https://nodesecurity.io/advisories/720 │ └───────────────┴──────────────────────────────────────────────────────────────┘ ┌──────────────────────────────────────────────────────────────────────────────┐ │ Manual Review │ │ Some vulnerabilities require your attention to resolve │ │ │ │ Visit https://go.npm.me/audit-guide for additional guidance │ └──────────────────────────────────────────────────────────────────────────────┘ ┌───────────────┬──────────────────────────────────────────────────────────────┐ │ Moderate │ Prototype Pollution │ ├───────────────┼──────────────────────────────────────────────────────────────┤ │ Package │ hoek │ ├───────────────┼──────────────────────────────────────────────────────────────┤ │ Patched in │ > 4.2.0 < 5.0.0 || >= 5.0.3 │ ├───────────────┼──────────────────────────────────────────────────────────────┤ │ Dependency of │ @juliusza/swaggerize-express │ ├───────────────┼──────────────────────────────────────────────────────────────┤ │ Path │ @juliusza/swaggerize-express > swaggerize-routes > enjoi > │ │ │ joi > topo > hoek │ ├───────────────┼──────────────────────────────────────────────────────────────┤ │ More info │ https://nodesecurity.io/advisories/566 │ └───────────────┴──────────────────────────────────────────────────────────────┘ found 12 vulnerabilities (10 moderate, 2 high) in 3306 scanned packages run `npm audit fix` to fix 7 of them. 3 vulnerabilities require semver-major dependency updates. 2 vulnerabilities require manual review. See the full report for details.
There are some good things and bad things this report illustrates.
- Great that the tool notes when there are major SEMVER (likely breaking) changes, like in the first one with eslint
- Illustrates how you get links to issues, hierarchy so you know how the package got imported
- If you follow the links, they are pretty empty (bad)
- Most link to github issues where you can see code (good!)
- The severity is questionable. When I look at any one of these, none of them scream that I MUST fix it now but there are 2 high and 10 moderate level items.
- The names of the issues are security focused. What is Prototype Pollution?
- Lots of these seem to come from eslint, which isn’t really part of the app when it runs, right?
- Note that it doesn’t include how old the issue is or how old the library is
My takeaway from all of these is that we really need to update to eliminate the prototype pollution and the code injection in the Morgan library because the combination of those can be bad. Generally code injection or code execution are bad news. I would also update the insufficient entropy while I was at it but that doesn’t scream that it must be done today given that I’m only using that as part of a SendGrid integration, so I probably wouldn’t do that or the others on their own if not for the code execution. Even still, chances are pretty high that I’m not vulnerable at all here.
I’d love to hear people’s comments about how I triaged these.
There are similar tools for other languages which we’ll list in the references. For the most part, the output is comparable. Some are better than others and some ecosystems are cleaner and more established than others.
This is a great thread on the Node security working group about the quality of how warnings are written really matters a lot.
Why It Matters
So … I talked about why it is difficult for developers to handle dependencies and then I walked through a detailed example which showed a lot of …
If I stop there, I’m not really doing the issue justice though.
The reason that this matters is that lots of open source (and closed source) libraries have vulnerabilities. In the Java world, Struts is the famous one partly because there have been a series of issues but also because there was a famous hack associated with them: Equifax.
It has also happened in Python for those that want to get on their high horses.
So we see backdoors being introduced to an ecosystem. Is there a backdoor in your application right now?
The other reason this matters is that when a vulnerability happens in libraries, or commonly packaged code, it is a good target for increased investment in exploits. Meaning, if I am trying to break things, libraries are a good target because they are used across a lot of organizations. Often, these vulnerabilities get weaponized into tools like metasploit such that they are trivially easy for an attacker with almost no knowledge to use.
So this is real.
There is kind of another thing to keep in mind here though. An example might be the best way to illustrate it.
A couple of years ago I was working with a team that built an app in Rails 3. If I remember correctly, they were using a Twitter gem in their app that had a vulnerability. They needed to update the gem. They also kind of wanted to move to Rails 4 but that involved a lot more work. The updated version of the Twitter gem was only compatible with Rails 4, not Rails 3.
To effectively update the dependency, they had two choices:
- Patch the Rails 3 gem themselves and use their altered gem
- Update to Rails 4, use the new gem and rewrite a bunch of their application
Neither of these is a good scenario, but the fact that they couldn’t easily update to Rails 4 meant that they essentially had to choose one of these options and put time into this instead of other work they had coming.
We see the same things with jQuery, Telerik, Spring, etc. The point is, if you let your app lag too far behind the current version, potentially even to an unsupported version you are inviting serious risks of issues. Again, there are two reasons:
- The older the library, the more likely it may be that there are known vulnerabilities in it
- The deeper the difference between the currently used and currently released API is work you may have to absorb
Let me say that just one more time another way to make sure it comes across. If we don’t update our applications, there could be an old vulnerable version of a library that we are using that we are forced to then update because of a terrible remote code execution issue (example) and to update we not only need to drop in the new version, we need to change code everywhere that it uses the library because method signatures may have changed, usage patterns may be completely different in a new major version that doesn’t have the vulnerability.
All of this is to say that there is a background reason why we need to stay updated anyway, and we should remember that.
Design A Process
As part of our SDLC, we should have a process for identifying vulnerabilities in our libraries and triaging them for updates. The identification process should include both tools and developers plugged into security distribution lists for the key technologies being used.
Typically, I like to recommend something like this:
|Tier||SLA||Process||Types of Vulnerabilities|
|1||2 hours||As fast as possible, branch from prod||Code Injection, SQL Injection, AuthZ Bypass|
|2||2 weeks||Put in sprint||XSS, Information disclosure|
|3||6 weeks||Put periodic update||Applying general updates, browser header thingies|
Note that the SLA’s and types are subjective and need to be tuned to your company and process. It may be that 6 weeks has to be a quarter at your organization. Be careful extending much beyond that because that opens a longer window of time you may be vulnerable.
Now, I know you just watched me explain that it is hard to update and here I am saying that you should update libraries just to stay up to date every 6 weeks (or your longest timeframe). I wrote another blog post about this too if you are interested.
Design Another Process
Yes, really. In addition to being able to identify and then triage issues as they come in and decide how important they really are, we should also be conscious of what we’re including in our applications.
Questions we should generally ask:
- Is a library actively maintained?
- How many committers are there?
- Do they have a track record of accepting pull requests?
- Is a library widely used?
- Does the library have a security response?
- Are we comfortable with the dependencies?
- What is the licensing?
- Is the library still in use by the application?
My personal observation is that in the Node world, people don’t do this very well.
You could even make an argument, and we have customers that do, that if security really is a prime objective, Node.js isn’t the right technology stack.
Jemurai produces code in Node.js so we’re not that far down the path but financial institutions we work with have some justification in taking that point of view.
Support the Process with Technology (Automate)
Process is great. Automation is also very helpful. If we run
npm audit every build we can’t miss that we’re on old dependencies. Getting the automation right is hard. It’s why we helped write OWASP Glue. It’s probably a deeper topic for another day. Some of this is already automated with GitHub security alerts.
One piece of advice we always give our clients related to dependency management, updating and tools is:
Design the process and use open source with automation to prove it works before you buy a commercial tool you think is going to drop in and solve your problem. Because they won’t. They may be an upgrade at a later time, but they don’t need to be the central foundation for your solution. By the way, if they are, you’ll never get to make the choice again without throwing away a lot of work.
Consider the following actions:
- Running a tool to detect vulnerabilities as part of your build
- Having a process for including new libraries and reviewing old ones
- Have a triage process so that when security issues are identified, you know how fast to respond
- Periodically stay up to date in general
Our own past posts:
Tools and other pertinent references: