Services
A common question came up again this week working with a developer (and friend) at a partner that does custom software development.
I’m working on a project that is implementing a bunch of Node microservices. What we are realizing is that we aren’t sure of the best way to manage package dependencies across multiple micro-services managed as different projects in bitbucket. So, we’re balancing trying to be reasonably current with our included packages to avoid security (and other) issues against the insanity of doing nothing but constantly updating packages across multiple projects, updating references to maintain consistency (as well as minimize the number of potential packages/versions that could have vulnerabilities), etc.
There really are a bunch of competing factors when we think about managing dependencies. Security folks generally tell developers to update immediately and developers resist with tones similar to the one in the quote above about “insanity of doing nothing but constantly updating packages”.
This post offers our most common advice on the topic.
First and foremost, let me step back and acknowledge that this is harder than it looks at first glance.
When we train on this, the developers in the room offer lots of reasons that it is challenging. Why is it hard? Well, in a java stack based application I might see 50-100 dependencies (without really trying). In Node, because of the proliferation of small packages, we often see thousands of dependencies. It is actually unsurprising in this context that there would be changes to a package every day.
Some changes are important, others are not. Just because a library has a serious vulnerability in it doesn’t mean that the code that uses the library exposes that vulnerability. If I’m using a database library and there is a vulnerability in methodXYZ() but I never call methodXYZ() then is my code vulnerable?
Not only that, often library updates introduce breaking changes that require code to be rewritten. I was bitten when Hibernate updated the default database key column type from using Integers to using Longs. I never thought to write unit tests that checked the types and we rolled an update to try to incorporate Hibernate updates and the whole system blew up in our face. We needed to have done more testing and/or manual QA to ensure that nothing under the hood of that update was going to break existing functionality. Similar things happen with libraries all the time. But many organizations can’t do more testing very easily.
Here’s an example of what we get when I run the current npm audit on an older version of our JASP API.
om:jasp-api mk$ npm audit
=== npm audit security report ===
# Run npm install eslint@5.14.1 to resolve 3 vulnerabilities
SEMVER WARNING: Recommended action is a potentially breaking change
┌───────────────┬──────────────────────────────────────────────────────────────┐
│ Moderate │ Prototype Pollution │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ Package │ lodash │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ Dependency of │ eslint │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ Path │ eslint > inquirer > lodash │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ More info │ https://nodesecurity.io/advisories/782 │
└───────────────┴──────────────────────────────────────────────────────────────┘
...
# Run npm install aws-sdk@2.406.0 to resolve 2 vulnerabilities<br>
┌───────────────┬──────────────────────────────────────────────────────────────┐
│ Moderate │ Prototype Pollution │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ Package │ lodash │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ Dependency of │ aws-sdk │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ Path │ aws-sdk > xml2js > xmlbuilder > lodash │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ More info │ https://nodesecurity.io/advisories/782 │
└───────────────┴──────────────────────────────────────────────────────────────┘
# Run npm install morgan@1.9.1 to resolve 1 vulnerability
┌───────────────┬──────────────────────────────────────────────────────────────┐
│ Moderate │ Code Injection │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ Package │ morgan │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ Dependency of │ morgan │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ Path │ morgan │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ More info │ https://nodesecurity.io/advisories/736 │
└───────────────┴──────────────────────────────────────────────────────────────┘
# Run npm update cryptiles --depth 5 to resolve 2 vulnerabilities
┌───────────────┬──────────────────────────────────────────────────────────────┐
│ High │ Insufficient Entropy │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ Package │ cryptiles │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ Dependency of │ @sendgrid/mail │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ Path │ @sendgrid/mail > @sendgrid/client > request > hawk > │
│ │ cryptiles │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ More info │ https://nodesecurity.io/advisories/720 │
└───────────────┴──────────────────────────────────────────────────────────────┘
┌──────────────────────────────────────────────────────────────────────────────┐
│ Manual Review │
│ Some vulnerabilities require your attention to resolve │
│ │
│ Visit https://go.npm.me/audit-guide for additional guidance │
└──────────────────────────────────────────────────────────────────────────────┘
┌───────────────┬──────────────────────────────────────────────────────────────┐
│ Moderate │ Prototype Pollution │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ Package │ hoek │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ Patched in │ > 4.2.0 < 5.0.0 || >= 5.0.3 │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ Dependency of │ @juliusza/swaggerize-express │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ Path │ @juliusza/swaggerize-express > swaggerize-routes > enjoi > │
│ │ joi > topo > hoek │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ More info │ https://nodesecurity.io/advisories/566 │
└───────────────┴──────────────────────────────────────────────────────────────┘
found 12 vulnerabilities (10 moderate, 2 high) in 3306 scanned packages
run `npm audit fix` to fix 7 of them.
3 vulnerabilities require semver-major dependency updates.
2 vulnerabilities require manual review. See the full report for details.There are some good things and bad things this report illustrates.
My takeaway from all of these is that we really need to update to eliminate the prototype pollution and the code injection in the Morgan library because the combination of those can be bad. Generally code injection or code execution are bad news. I would also update the insufficient entropy while I was at it but that doesn’t scream that it must be done today given that I’m only using that as part of a SendGrid integration, so I probably wouldn’t do that or the others on their own if not for the code execution. Even still, chances are pretty high that I’m not vulnerable at all here.
I’d love to hear people’s comments about how I triaged these.
There are similar tools for other languages which we’ll list in the references. For the most part, the output is comparable. Some are better than others and some ecosystems are cleaner and more established than others.
This is a great thread on the Node security working group about the quality of how warnings are written really matters a lot.
So … I talked about why it is difficult for developers to handle dependencies and then I walked through a detailed example which showed a lot of …
Meh.
If I stop there, I’m not really doing the issue justice though.
The reason that this matters is that lots of open source (and closed source) libraries have vulnerabilities. In the Java world, Struts is the famous one partly because there have been a series of issues but also because there was a famous hack associated with them: Equifax.
In the Node world, there have been a few cases 1,2,3 where malicious folks buried backdoors in libraries that were commonly used.
It has also happened in Python for those that want to get on their high horses.
So we see backdoors being introduced to an ecosystem. Is there a backdoor in your application right now?
The other reason this matters is that when a vulnerability happens in libraries, or commonly packaged code, it is a good target for increased investment in exploits. Meaning, if I am trying to break things, libraries are a good target because they are used across a lot of organizations. Often, these vulnerabilities get weaponized into tools like metasploit such that they are trivially easy for an attacker with almost no knowledge to use.
So this is real.
There is kind of another thing to keep in mind here though. An example might be the best way to illustrate it.
A couple of years ago I was working with a team that built an app in Rails 3. If I remember correctly, they were using a Twitter gem in their app that had a vulnerability. They needed to update the gem. They also kind of wanted to move to Rails 4 but that involved a lot more work. The updated version of the Twitter gem was only compatible with Rails 4, not Rails 3.
To effectively update the dependency, they had two choices:
Patch the Rails 3 gem themselves and use their altered gem
Update to Rails 4, use the new gem and rewrite a bunch of their application
Neither of these is a good scenario, but the fact that they couldn’t easily update to Rails 4 meant that they essentially had to choose one of these options and put time into this instead of other work they had coming.
We see the same things with jQuery, Telerik, Spring, etc. The point is, if you let your app lag too far behind the current version, potentially even to an unsupported version you are inviting serious risks of issues. Again, there are two reasons:
The older the library, the more likely it may be that there are known vulnerabilities in it
The deeper the difference between the currently used and currently released API is work you may have to absorb
Let me say that just one more time another way to make sure it comes across. If we don’t update our applications, there could be an old vulnerable version of a library that we are using that we are forced to then update because of a terrible remote code execution issue (example) and to update we not only need to drop in the new version, we need to change code everywhere that it uses the library because method signatures may have changed, usage patterns may be completely different in a new major version that doesn’t have the vulnerability.
All of this is to say that there is a background reason why we need to stay updated anyway, and we should remember that.
As part of our SDLC, we should have a process for identifying vulnerabilities in our libraries and triaging them for updates. The identification process should include both tools and developers plugged into security distribution lists for the key technologies being used.
Typically, I like to recommend something like this:
Tier | SLA | Process | Types of Vulnerabilities |
|---|---|---|---|
| 1 | 2 hours | As fast as possible, branch from prod | Code InjectionInjection is a type of cyber attack where an attacker exploits vulnerabilities in an application or system to inject malicious code or commands. Injection attacks are often carried out by inserting code or commands into input fields or parameters that are passed to an application or database., SQL Injection, AuthZ Bypass |
| 2 | 2 weeks | Put in sprint | XSS, Information disclosure |
| 3 | 6 weeks | Put periodic update | Applying general updates, browser header thingies |
Note that the SLA’s and types are subjective and need to be tuned to your company and process. It may be that 6 weeks has to be a quarter at your organization. Be careful extending much beyond that because that opens a longer window of time you may be vulnerable.
Now, I know you just watched me explain that it is hard to update and here I am saying that you should update libraries just to stay up to date every 6 weeks (or your longest timeframe). I wrote another blog post about this too if you are interested.
Yes, really. In addition to being able to identify and then triage issues as they come in and decide how important they really are, we should also be conscious of what we’re including in our applications.
Questions we should generally ask:
My personal observation is that in the Node world, people don’t do this very well.
You could even make an argument, and we have customers that do, that if security really is a prime objective, Node.js isn’t the right technology stack.
Jemurai produces code in Node.js so we’re not that far down the path but financial institutions we work with have some justification in taking that point of view.
Process is great. Automation is also very helpful. If we run npm audit every build we can’t miss that we’re on old dependencies. Getting the automation right is hard. It’s why we helped write OWASP Glue. It’s probably a deeper topic for another day. Some of this is already automated with GitHub security alerts.
One piece of advice we always give our clients related to dependency management, updating and tools is:
Design the process and use open source with automation to prove it works before you buy a commercial tool you think is going to drop in and solve your problem. Because they won’t. They may be an upgrade at a later time, but they don’t need to be the central foundation for your solution. By the way, if they are, you’ll never get to make the choice again without throwing away a lot of work.
Consider the following actions:
Boosh! Enjoy!
Our own past posts:
Tools and other pertinent references:
