Tag Archive glue

Live Coding a Glue Task at AppSecUSA – Video

Matt Konda No Comments

Here is the video from the Glue and live coding talk at AppSecUSA.

Live Coding a New Glue Task at AppSecUSA

Matt Konda No Comments

At AppSecUSA, OWASP Glue, a project we contribute heavily to, was asked to present in the project showcase.  I put together an overview talk about how the tool is architected and what we use it for.  Then, we added a new task live during the talk.  I thought that was enough fun that it was worth blogging about.  I would like to start by acknowledging Omer Levi Hevroni, of Soluto, who has also contributed and continued support for Glue over the past year +.

Why Glue?

We started working on Glue because we found that most security tools don’t integrate all that cleanly with CI/CD or other developer oriented systems.  Our goal is always to get the checking as close to the time of code authorship as possible.  Further, we thought that having it wrap open source tools would make them more accessible.  Philosophically, I’m not that excited about tools (see earlier post) but we should definitely be using them as much as we can provided they don’t get in our way.  Ultimately, we wrote Glue to make it easy to run a few security tools and have the output dump into Jira or a CSV.

Glue Architecture

So we defined Glue with the following core concepts:

  • Mounters – these make the code available.  Could be pulling from GitHub or opening a docker image.
  • Tasks:  Files – these analyze files.  Examples are ClamAV and hashdeep.
  • Tasks:  Code – these are some level of code analysis.  Some are first class static tools, others are searching for dependencies or secrets.
  • Tasks:  Live – these are running some kind of live tool like Zap against a system.
  • Filters – filters take the list of findings and eliminate some based on whatever the filter criteria are.  An example is limiting ZAP X-Frame-Options findings to 1 per scan instead of 1 per page.
  • Reporters – reporters take the list of findings and push them wherever you want them to end up.  JIRA, TeamCity, Pivotal Tracker, CSV, JSON, etc.

Live Coding

OK, so a common task is to say

“Hey, this is cool but I want to do this with a tool that Glue doesn’t yet support.”

Great.  That should be easy.  Let’s walk through the steps.  We’ll do this with a python dependency checker:  safety.

Set Up the Task

A good way to start a new task is to copy an existing one.  In this case, we’re going to do a python based task so let’s copy the bandit task and alter to match:

require 'glue/tasks/base_task'  # We need these libraries.  The base_task gives us a report method.
require 'json'
require 'glue/util'
require 'pathname'

# This was written live during AppSecUSA 2018.
class Glue::SafetyCheck < Glue::BaseTask  # Extend the base task.

Glue::Tasks.add self  # Glue is dynamic and discovers tasks based on a static list.  This adds this task to the list.
includeGlue::Util

def initialize(trigger, tracker)
  super(trigger, tracker)
  @name="SafetyCheck"                             # This is the name of the check.  -t safetycheck (lowered)
  @description="Source analysis for Python"
  @stage= :code                                   # Stage indicates when the check should run.
  @labels<<"code"<<"python"                       # The labels allow a user to run python related tasks.  -l python
end

Now we have the start of a class that defines our SafetyCheck task.

Run the Tool :  Safety Check

Here we just need to implement the run method and tell the task how to call the tool we want to run.  In this case, we want it to create json.  The resulting code is:

def run
  rootpath = @trigger.path
  @result=runsystem(true, "safety", "check", "--json", "-r", "#{rootpath}/requirements.txt")
end

The @trigger is set when the Task is initialized (see above) and includes information about where the code is.  We use that to know where the path that we want to analyze is.  Then we use one of the provided util methods runsystem to invoke safety check with the parameters we want.

Note that we are putting the result in the @result instance variable.

Parse the Results

Once the tool runs, we have the output in our @result variable.  So we can look at it and parse out the JSON as follows:

 def analyze
    puts @result
    results = clean_result(@result)
    begin
      parsed = JSON.parse(results)
      parsed.each do |item|  
        source = { :scanner => @name, :file => "#{item[0]} #{item[2]} from #{@trigger.path}/requirements.txt", :line => nil, :code => nil }
        report "Library #{item[0]} has known vulnerabilities.", item[3], source, severity("medium"), fingerprint(item[3]) 
      end 
    rescue Exception => e
      Glue.warn e.message
      Glue.warn e.backtrace
      Glue.warn "Raw result: #{@result}"
    end
  end

Here, we call a clean_result method on the result first.  You can look here for detail, but it is just pulling the warnings that the tool emits that make the output not valid JSON.  We do this to make sure the JSON is parseable for our output.  This is a common issue with open source tools.  I don’t underestimate the value of making these things just work.

The magic here is really in the two lines that set the source and then report it.  The source in Glue terms is the thing that found the issue and where it found it.  In this case, we’re setting it to be our Task (SafetyCheck) and the library name in the output from file containing the dependencies. (requirements.txt)

The report method is supplied by the parent BaseTask class and takes as arguments:  description, detail, source, severity and fingerprint.

def report description, detail, source, severity, fingerprint
    finding = Glue::Finding.new( @trigger.appname, description, detail, source, severity, fingerprint, self.class.name )
    @findings << finding
end

You can see if you look closely that we set all of the severities to Medium here because safety check doesn’t give us better information.  We also use the supplied fingerprint method to make sure that we know if we have a duplicate.  You can also see that the result of calling report is that we have a new finding and the finding is added to the array of findings that were created by this task.  We get the trigger name, the check name, a timestamp, etc. just by using the common Finding and BaseTask classes.

Making Sure the Tool is Available

In addition to run and analyze the other method we expect to have in our Task is a method called supported?.  The purpose of this method is to check that the tool is available.  Here’s the implementation we came up with for safety check, which doesn’t have a version call from the CLI.

 def supported?
    supported=runsystem(true, "safety", "check", "--help")
    if supported =~ /command not found/
      Glue.notify "Install python and pip."
      Glue.notify "Run: pip install safety"
      Glue.notify "See: https://github.com/pyupio/safety"
      return false
    else
      return true
    end
  end

The idea here is to run the tool in a way that tests if it is available and alerts the user if it is not.  Graceful degredation as it were…

The Meta

Here is the code from the Tasks ruby file that runs all the tasks.

if task.stage == stage and task.supported?
  if task.labels.intersect? tracker.options[:labels] or      # Only run tasks with labels
    ( run_tasks and run_tasks.include? task_name.downcase )  # or that are explicitly requested.

    Glue.notify "#{stage} - #{task_name} - #{task.labels}"
    task.run
    task.analyze
    ...

Here you see the supported?, run and analyze methods getting called.  You also see the labels and tasks being applied.  Its not magic, but it might look weird when you first jump in.

Conclusion

We wanted to create a new task.  We did it in about 20 minutes in a talk.  Of course, I wrote the code around it so it was easy.  But it does illustrate how simple it is to add a new task to Glue.

If you want to see this check in action, you can run something like:

bin/glue -t safetycheck https://github.com/DefectDojo/django-DefectDojo

We’re getting the Docker image updated, but once available you can run:

docker run owasp/glue -t safetycheck https://github.com/DefectDojo/django-DefectDojo

We hope people can digest and understand Glue.  We think it is an easy way to get a lot of automation running quickly.  We even run a cluster of Kubernetes nodes doing analysis.  This in addition to integrating into Jenkins builds or git hooks.  Happy Open Source coding!!!

Glue 0.9.4 and Scout2

Matt Konda No Comments

Glue

We spend a fair amount of time building and using OWASP Glue to improve security automation at clients.  The idea is generally to make it easy to run tools from CI/CD (eg.  Jenkins) and collect results in JIRA.  In a way, Glue is like ThreadFix or other frameworks that collect results from different tools.  Recently, we thought it would be cool to extend some of what we were doing to AWS.  We have our own scripts we use to examine AWS via APIs but we realized that Scout2 was probably ahead of us and it would be a good place to start.

Scout2

The fine folks at NCCGroup wrote and open sourced a tool for inspecting AWS security called Scout2.  You can use it directly, and we recommend it, based on the description here:  https://github.com/nccgroup/Scout2.  It produces an HTML report like this:

For most programmers, running Scout2 is easy.  It just requires a little bit of python setup and an appropriate AWS profile.  So it wasn’t so much the barrier to entry that made us want to integrate it into Glue so much as the idea that we could take the results and integrate them into the workflow (JIRA) that we are using for other findings from other tools.  We thought that having an easy way to pull the results together and publish them based on Jenkins would be pretty useful.

What’s Coming with Glue

Glue has been a fun project that we’ve used opportunistically.  The next set of goals with Glue is to clean it up, improve tests and documentation and prepare for a legitimate 1.0 release.  At that point, we’ll probably also try to get Glue submitted for Lab status as an OWASP project.

Automate All The Things

Matt Konda No Comments

Today I gave a talk at a company’s internal security conference about automation.  The slides are on speakerdeck.  A video is on Vimeo.

The point of the talk was threefold:

  1. Explain where automation works well and examples of where we use it with OWASP Glue
  2. Explain newish cool automation like cloud analysis and pre-audit preparation
  3. Talk about how really, automation can only get us so far because we need the interaction and communication to fix things

I’d be interested to hear feedback!

 

Glue Update

Matt Konda No Comments

There have been several recent improvements with Glue.  Its been awesome to have more people committing to the project and adding in different ways.

One is related to ZAP integration, which is finally getting more of the attention it needs.  Another is related to reporting to JIRA.  Still another is a way to fail builds only on certain thresholds of errors.  We have also been working on integrations for Contrast and Burp.  We’ve added a more representative Jenkins Build Pipeline integration example.

We added support to search for entropy in passwords via TruffleHog.

What would you like to see in Glue?  Where do you think we need to be to get to a credible 1.0?

Glue 0.9.3

Matt Konda No Comments

Introduction

At Jemurai, we contribute extensively to OWASP Glue and use it on some of our projects where it makes sense to tie together automation around security.  We kept seeing the same types of integration challenges and found that it was useful to have a common starting point to solve them.  It is far from perfect and we would refer people to alternatives like ThreadFix and OWTF

What is Glue?

Glue is basically a Ruby gem (library) that knows how to run a variety of security tools, normalize the output to a set structure and then push the output to known useful places like Jira.  We package Glue in a docker image to try to make it easy to set up all the different important moving parts (eg. Java, Python, Ruby and tools).  You can get and run Glue from docker as easy as:

docker run owasp/glue:0.9.3

Or, for a more helpful example, we can run brakeman and get the output as follows:

docker run --rm owasp/glue:0.9.3 -t brakeman https://github.com/Jemurai/triage.git

Architecture

The idea behind Glue is to be able to process different types of files via Mounters.  Then to be able to analyze using different Tasks, filter with different Filters and report with different Reporters (CSV, Jira, Pivotal, etc.).  Ultimately, there are the concept of stages that can be easily extended.  The reason I’m writing the post today is because I wanted to add a Bandit task and it was so easy that all I had to do was add this one file:  https://github.com/OWASP/glue/blob/master/lib/glue/tasks/bandit.rb.

When I was done, you could run bandit from the Glue docker image and push results to Jira or anywhere else:

docker run --rm owasp/glue:0.9.3 -t bandit https://github.com/humphd/have-fun-with-machine-learning.git

The following diagram illustrates the stages of the pipeline of functions that Glue performs.

Check out Glue or reach out if you want to talk about some of the common challenges in security automation.