Engineering

Announcing “Inside Out” Tech Talks

Matt Konda No Comments

As a small, growing and disruptive company we place a major focus on training our employees.

We’ve tried a lot of different things:  Capture the Flag games, internal videos, weekly tech talks, etc.  It’s an ongoing challenge and a continually improving process.  In a recent team discussion, we realized that there might be an interesting value to making some of those tech talks public.  It’s a way for us to provide something valuable to our community while giving our team a platform to present and cross training on technology and software security problems we’re facing.  For example, we’re seeing Hashicorp Vault and Marathon at Client X, or we’re using OWASP Glue with Jenkins at Client Y.

Somehow we came up with the idea of Inside Out Tech Talks, where we take one of our regular tech talks and make it open to the public.

The first will be 12/13 at 1:00 PM CST.

Join us on Zoom:  https://zoom.us/meeting/register/cd9408314686923e7510d14dfea9e911.

The topic is Security Automation.

 

Thinking About Secrets

Matt Konda No Comments

Introduction

We have two types of projects that often uncover secrets being shared in ways that aren’t well thought through.

  1. During code review, it is actually rare that we do not find some sort of secret.  Maybe a database password or maybe an ssh key.  Sometimes, it is AWS credentials.  We’ve even built our own code review assist tool to check for all the ones we commonly see.
  2. During security engineering or appsec automation projects we end up wiring static analysis tools to source code and JIRA and this often uncovers plaintext secrets in git.

So, generally, plaintext secrets are everywhere.

Best Practice

I said above that people are sharing secrets in ways that aren’t well thought through.  Let me expand on that.  If all of your developers have access to secrets, that’s a problem.  Of course, most developers aren’t going to do anything nefarious but they might, especially after they leave.  Most companies have a further challenge that it is very difficult to change system passwords.  So .. developer leaves the company and chances are low any secrets are changing.  Suppose a laptop with source code on it gets stolen?

The other problem with having secrets around is that it makes it easy for an attacker to pivot and find other things they can target.  Suppose I get code execution on Server 1.  If all of the secrets Server 1 uses are stored in files on the server that the code uses, it makes that pivot to get on Server 2 via SSH or pull data from DB Server 3 trivial.

Testing

Here are two instant ways to look for secrets in your code:

docker run owasp/glue -t sfl https://github.com/Jemurai/triage.git

This runs a check for sensitive files that are often in the source code.

docker run owasp/glue -t trufflehog https://github.com/Jemurai/triage.git

This looks for entropy in the files in a project.  It can take a while to run but is a good way to find things like keys or generated passwords.

Nothing beats a person that is looking carefully and knows what to look for but grep is your friend for sure.  We try to find these and offer alternatives for storing secrets.

The Alternatives

Generally, we want to keep secrets in a place where:

  • People can’t just go read them
  • It is easy to change them
  • We know if anyone uses them (audit)

We see a lot of projects adopting Vault and tools like it to store secrets.

Even better is a scenario where credentials don’t even exist but get generated for a particular action and then automatically expired.  Ideally, we require MFA.  99-Design’s AWS-Vault does this with AWS and its sessions in an elegant way.  This pattern, in general, allows us to know that there aren’t existing passwords out there that people could use without our necessarily realizing.  It also reduces the challenge of responding to a stolen laptop for example.

References

An older reference from a ThoughtWorker:  https://danielsomerfield.github.io/turtles/

A tool:  https://www.vaultproject.io/

Another tool:  https://github.com/99designs/aws-vault

An upcoming Jemurai Tech Talk:  https://www.jemurai.com/webinar/3topopensourcetoolsforsecretsmanagement

Glue 0.9.4 and Scout2

Matt Konda No Comments

Glue

We spend a fair amount of time building and using OWASP Glue to improve security automation at clients.  The idea is generally to make it easy to run tools from CI/CD (eg.  Jenkins) and collect results in JIRA.  In a way, Glue is like ThreadFix or other frameworks that collect results from different tools.  Recently, we thought it would be cool to extend some of what we were doing to AWS.  We have our own scripts we use to examine AWS via APIs but we realized that Scout2 was probably ahead of us and it would be a good place to start.

Scout2

The fine folks at NCCGroup wrote and open sourced a tool for inspecting AWS security called Scout2.  You can use it directly, and we recommend it, based on the description here:  https://github.com/nccgroup/Scout2.  It produces an HTML report like this:

For most programmers, running Scout2 is easy.  It just requires a little bit of python setup and an appropriate AWS profile.  So it wasn’t so much the barrier to entry that made us want to integrate it into Glue so much as the idea that we could take the results and integrate them into the workflow (JIRA) that we are using for other findings from other tools.  We thought that having an easy way to pull the results together and publish them based on Jenkins would be pretty useful.

What’s Coming with Glue

Glue has been a fun project that we’ve used opportunistically.  The next set of goals with Glue is to clean it up, improve tests and documentation and prepare for a legitimate 1.0 release.  At that point, we’ll probably also try to get Glue submitted for Lab status as an OWASP project.

Signal, Audit and Logging – Introduction

Matt Konda No Comments

At clients, we work to make sure the best information is available to:

  1. Debug an application
  2. Track what happens in an application
  3. Produce security signal for monitoring

Often, developers or security folks think of these as overlapping.  We hear:  “We’re using Log4J wrapped with SLF4J, it seems really redundant to do anything else.”

In practice, we believe the information is different and needs to be stored and reviewed in different ways by different people.  That’s why we build libraries to help integrate these things into applications easily – each as a first class piece of information.  As we examine each in further detail, we’ll call out the technology, audience involved and typical content.

Logging

Let’s start with logging because every developer knows about logging, right?  We work with some companies that log every request that they process.  That seems like a lot and should start to trigger alarm bells about what information lives in the logs – but let’s not be mad at logging.  For the most part, there are established solutions for doing it.  The logs need to get aggregated or centralized somewhere and then we can try to see what happened.

We would be remiss here not to point out that it is really important to keep sensitive data out of logs.  We’ve seen everything from card numbers to passwords to reset tokens to session ids …

But the point is, there isn’t anything wrong with a little log.debug(“XYZ”); or log.warn(“Data is stale”);.  From a maintenance and debugging perspective, this information is valuable – generally to operations.

Technology:  Typically file based, then aggregated.  Need text search.  High volume.  Retained for relatively short periods of time (weeks).

Audience:  Developers, Operations

Content:  Freeform – anything a developer might think is useful.

Audit

Some applications explicitly need to be able to produce an audit record for the objects they manage.  This might be who created it, when it changed and how – at who’s direction.  It might even be who accessed this data?  Consider the Stripe interface where they let you access your secret.  The secret is obscured and you have to take an extra action to see it.  Pretty sure they audit that so they know who saw it when.

Technically, you could write audit messages to logs.  This results in tedious work getting the detail back out of the logs and in any system where logs are not smoothly aggregated or can’t be managed at scale, this approach falls down.  Furthermore, someone looking for the messages needs to sift through lots of unrelated data.

A deeper issue is that if you want to produce a true audit record, like for a partner or a customer or an auditor, you can’t just give them all your dev logs!  We want to be able to produce a report tailored for the question they are asking and containing only data for the users they should be able to ask about.  Also, audit records need to be stored for a lot longer than log messages.

Technology:  Queryable interface, centralized long term storage, retained “for a long time”

Audience:  Compliance, Partners, Auditors

Content:  Specific object reads and changes (ideally with before / after visibility) associated to users & time.

Two deeper notes here:

  1. There is a lot more that you can do with structured auditing to be proactive, but that is Aaron Bedra magic so we’ll leave that to him to describe in some venue of his choosing.
  2. Audit information is richer and more valuable the closer it is to actions.  So we discourage general filters that indicate which endpoint got called, for example, because it can’t provide the rich context that a deeper integration can.

Signal

When I say signal, I really mean security signal.  As in, the opposite of noise.  Let’s face it, most of what is in logs is noise.  Even cutting edge technology built to collect and analyze logs produces a ton of noise.  When we get into the application and signal specific events there, we can break out of the noise and give security monitoring teams lots of rich data.

For example, we may want to know about failed logins.  A stream of failed logins looks bad.  A similar stream followed by a successful login looks worse.  (Exercise for reader)  Either way, this is information the security team probably doesn’t see right now.  Go a step deeper – what if input validation fails?  What if someone tries to do something they shouldn’t have permission to do?  Should security get notified?  Obviously the goal would be to take some defensive actions autonomously and in systems we’ve worked on, this works best when you can capture the actual events you care the most about.  Where can you do that?  In the application.

Another key thing with Signal is that it needs to go to the right place.  Often security operations teams are using their own SEIM that is different than the log collector used by developers.  That is smart.  They are optimized for different things.  But we need to help developers get the security events to the SEIM.

Technology:  Push signal to syslog + SEIM, ideally not retained for more than weeks but aggregated and processed for future context.

Audience:  Security Operations Team (The Watchers), Automated Security Response

Content:  Specific security events only.

Our Conclusion

At companies that have the capability and resources (say they have compliance and security monitoring teams) separating these application generated log stream messages has value because they are used by different people for different things in different tools.

We may circle back in the future with another post about our libraries for doing these things and some of the more extended benefits or specific examples of data to track.  Let us know if you are interested!