We had a customer ask us to dig for some indicators of compromise in their AWS account. We are already using our JASP tool to help them to check security configurations in general, so we took the opportunity to formalize some of what we’re doing into a tool which we plan to open source once we clean it up. This post presents some of the types of things that are challenging to just check in JASP and how we’re thinking about the tool.
The Limits of JASP
It may help to start by explaining what JASP does and doesn’t do. It does not look at actual CloudTrail events. It also doesn’t help an organization to identify users that are provioned that shouldn’t be. It looks for things like weak IAM password policies or other configuration issues. JASP is invaluable in detecting things like ports open to the world that are in new VPC’s. We think it makes a lot of sense to use it to kind of keep tabs on your AWS environment.
But in this case, we’re looking for contextual data that would suggest something unusual is happening. Its kind of a different case.
So What Are We Looking For?
Generally we’re looking for anything that suggests unusual behavior. Suppose an internal engineer were misusing the environment or one of our admins got compromised
Note that while many people use the AWS Console (The web GUI), we heavily focus on using the API’s and recommend clients do as well because the UI is so rich with data that it can be tricky to be sure that you’re looking at all of the relevant information. Also, a script to click through x,y,z different ways doesn’t scale (or allow repeated quick usage and tweaking) the same way a script that uses an API does.
General AWS API’s
Some examples of the things we’re looking for didn’t even come from cloudtrail. They were accessible directly from other AWS API’s eg:
- New users - any provisioned user should be mappable to a known reason.
aws-vault exec jemurai -- aws iam list-users --output text | cut -f 7 > iam_users.txt
- Changes in user’s groups or roles should be mappable to a ticket.
aws iam list-groups --output text
aws iam list-roles
- Access keys should be predictably used. Creating or changing access keys seems weird. (See snippet below)
- Want to see local IAM policies
aws iam list-policies --scope Local --output text | cut -f 4,9,10
- Want to see new VPC’s, gateways, subnets:
aws ec2 describe-vpcs --output text
aws ec2 describe-vpn-gateways
aws ec2 describe-subnets --output text
- New buckets
aws s3api list-buckets --output text
Here is a quick script to check user access keys:
aws iam list-users --output text | cut -f 7 > iam_users.txt allUsers=$(cat ./iam_users.txt) for userName in $allUsers; do aws iam list-access-keys --user-name $userName --output text | cut -f 3,4,5 done
What To Look for in CloudTrail
Our CloudTrail tool ended up looking for:
- Specific access keys that were created in a window of time we care about.
- StopLogging events which would stop other things from showing up in CloudTrail. We want to see 0 of these.
- ConsoleLogin events
- We want to discourage this in general but also know who does it.
- Related to network configuration:
There are a lot more events we can specifically look for. This is just an initial taste.
We stumbled across the GorillaStack slack bot (see references) that can help you keep track of events as they go by. That’s a neat approach.
Certainly using CloudWatch to create alerts on certain things makes sense.
Our approach was to write a Go program to collect a bunch of the events and data for view. We’re imagining that DevOps teams will want to modify this or adapt it to their use case. We also see the tool as finding candidate data that someone has to review. In theory, this could get incorporated into a UI or tool where these events could be filtered or responded to. For us, having a simple way to slice in to the data was the starting point.
We’re trying to figure out the best way for the tool to grab these and make them accessible to the right people in an organization. Should this be run through CI/CD as a nightly thing? Should it be hourly? We’d like to get the tool to be easy to set up as a Lambda. Once we have this set up as a first class Jemurai Open Source project we’ll be able to discuss it on Gitter and point to it on GitHub. Now we just need time!
This was a foray into Go and more dynamic AWS management. I expect we’ll continue to build custom tooling around AWS in Go now.
Our AWS cloud tool:
Tools and other pertinent references:
Want to stay up to date with the lastest from Jemurai?
Sign up for our monthly newsletter!