Pipeline Security Automation

This post talks about how we approach security automation in BitBucket Pipelines. It also introduces some new open source tools we built and use in the process.

Security In Pipelines

We’ve written before about using GitHub Actions and provided an Action friendly “workflow” with our Crush tool.

At a high level, Pipelines and Actions just do some computing work for you in Atlassian or Github’s data center. Often that work is related to source code, testing, or deployment.

Both leverage containers heavily for sandboxing the work that happens in the build and provide some level of abstraction that you can use to build your own pipes or workflows.

On some level, we’re using them to do the same work we used to use Jenkins or CircleCI or Travis to do.

We like automating security in Pipelines or Actions because it makes it easy to build security into your natural development workflows.

Real World

So what are we really talking about? What do we want to have happen?

We want to be able to apply any automated tools flexibly in a controlled environment that already has our code artifacts and knows about the key events.

Triggers for actions and pipelines can be:

• Push to a branch (we often set up a securityautomation/* branch to make this very deliberate at first)
• Push to master or release (i.e. when we deploy)
• A cron based schedule

The tools we want to run may range from:

• Secrets detection
• Vulnerable dependency detection
• Code smells
• Dynamic scanning (ZAP)
• Cloud configuration checks

Of course, we can run security unit tests and trigger other integrations (eg. SonarCloud) as well.

Just remember:

Tools are necessary but must be used by skilled operators.

I can’t tell you how often I see security tools installed but not used effectively.

OK Show Us Code

Here is an example of how we have a pipeline configured:

pipelines:
  branches:
    '{securityautomation/*}':
    - parallel:
      - step:
          name: Run Crush
          image: golang:1.16
          script:
            - go get -u github.com/jemurai/crush@v1.0.5
            - crush examine --threshold 7 . > crush.json
          artifacts:
            - crush.json
      - step:
          name: Dep Check
          image: openjdk:8
          script:
            - wget https://github.com/jeremylong/DependencyCheck/releases/download/v6.1.5/dependency-check-6.1.5-release.zip
            - unzip dependency-check-6.1.5-release.zip
            - rm dependency-check-6.1.5-release.zip
            - ./dependency-check/bin/dependency-check.sh --failOnCVSS 6 --exclude **/dependency-check/**/*.jar -f JSON --prettyPrint --scan .
          artifacts:
            - dependency-check-report.json
    - step:
        name: Report
        image: golang:1.16
        script:
          - go get -u github.com/jemurai/depcheck2off@v1.0.0
          - go get -u github.com/jemurai/off2jira@v1.0.0
          - depcheck2off ./dependency-check-report.json > ./depcheck.off.json
          - off2jira ./depcheck.off.json
          - off2jira ./crush.json

Let’s walk through it and talk about what is happening.

First, the branches part tells BitBucket when to run the pipeline. In this case, it will be on any push to a branch under securityautomation.

branches:
    '{securityautomation/*}':

We like doing this because it helps to isolate your security related chnages and ensures that what you are finding doesn’t break other builds. In the long run, we want to have security tooling run more often.

Then we need to understand that there are three steps defined in the pipeline:

• Crush
• Dep Check
• Report

Crush and Depenendency Check are both code analysis so they can run in parallel. Hence the parallel: before their step: definitions.

To run Crush, we pull a base golang image image: golang:1.16, install Crush and run it. We drop the output into an artifact that means it will be available later.

- step:
    name: Run Crush
    image: golang:1.16
    script:
        - go get -u github.com/jemurai/crush@v1.0.5
        - crush examine --threshold 7 . > crush.json
    artifacts:
        - crush.json

Running Dependency Check is similar. You can see that we’re pulling a release from GitHub and unzipping it. This is on an openjdk image. Then we invoke dependency check and put the report in an artifact.

- step:
    name: Dep Check
    image: openjdk:8
    script:
        - wget https://github.com/jeremylong/DependencyCheck/releases/download/v6.1.5/dependency-check-6.1.5-release.zip
        - unzip dependency-check-6.1.5-release.zip
        - rm dependency-check-6.1.5-release.zip
        - ./dependency-check/bin/dependency-check.sh --failOnCVSS 6 --exclude **/dependency-check/**/*.jar -f JSON --prettyPrint --scan .
    artifacts:
        - dependency-check-report.json

The next part “Report” is interesting and we’re going to put it in a whole new section.

Reporting and Rethinking Security Tooling

Once we have Crush and Dependency Check output, we want to do something with it. We could leave it in BitBucket as an artifact and refer to the plain text file. That is better than not running the tools, but we also want make these visible and integrate into our normal processes.

Here’s how that looks in the pipeline we defined where we’re pushing the issues identified by Crush and OWASP Dependency Check to JIRA:

- step:
    name: Report
    image: golang:1.16
    script:
       - go get -u github.com/jemurai/depcheck2off@v1.0.0
       - go get -u github.com/jemurai/off2jira@v1.0.0
       - depcheck2off ./dependency-check-report.json > ./depcheck.off.json
       - off2jira ./depcheck.off.json
       - off2jira ./crush.json

Here we are installing and using two new golang based tools:

• depcheck2off - which takes dependency check output and puts it in OWASP off format.
• off2jira - which takes a file of off findings and pushes them to JIRA.

The basic philosophy is to build small tools that do one thing and do it in a simple and predictable way. This goes directly against our own past approach with OWASP Glue which we retired.

With Glue, you could run different tools and push the output to a variety of trackers. The problem was that you ended up with a JVM, Python, Ruby, Node and an ever growing docker image. That made it hard to incorporate into pipelines efficiently. We also had to maintain everything and keep everything working to get an update pushed. It was a monolith.

With the Jemurai autom8d set of tools, we’re taking more of a classic Unix philosophy and building small purpose built utilities that can be put together in a number of ways.

So far we have:

• fkit
• depcheck2off
• off2jira

We already have plans to build:

• off2csv
• zap2off

We also want to adapt and integrate some other code we have that does an inventory and metrics across repositories.

We’d love to hear from you about others that would be useful! We can help with this type of automation while doing so with open tooling you can leverage for the long term.

Leverage Pipelines

The great thing about pipelines (and actions) is that once you understand them, you can effectively push security tooling to a large number of projects quite easily.

Note that there are compute charges associated with running pipelines (or actions).

We have also had good success helping companies who leverage BitBucket or GitHub cloud because we can actually help commit the code that starts the autoamtion project off. Combined with some training and a retained ongoing support setup - we can enable clients to very quickly improve their internal app security posture.

References

• Pipelines
• Crush GitHub Action
• GitHub Actions
• Pipes
• Workflows
• OFF
• Crush
• depcheck2off
• off2jira
• fkit