Large Scale Vulnerability Scanning with Jenkins

Find Security Bugs can often uncover interesting findings that may lead to the discovery of critical vulnerabilities. Back in May, we published on this blog two vulnerabilities in components of Spring, a Java web framework, using this tool. However, the process of using Find Security Bugs can be a little bit tedious to unseasoned Java users. Also, the process of analyzing compiled code and triaging the findings needed improvements. Here is the solution that was built to find vulnerabilities at scale.

The Problem

If you are developing a Java project, the integration of Spotbugs in Maven should be straightforward. If your goal is to analyze Java binaries to find vulnerabilities, it is a different challenge. First, you must run Find Security Bugs on the Java binaries. The output of this first step is a report of the warnings. This report can be in multiple formats such as HTML or XML. In order to validate the exploitability of the warnings, the researcher has to look at the source code. To obtain the source code, we can use a Java decompiler such as JD or CFR.

Now with the report and the code, we can analyze the code paths of the project and find vulnerabilities. For each finding, we can inspect the corresponding file and find where the warning is raised. At this point, we can inspect the code path to determine the exploitability. As you can tell, there’s a couple of steps required and the resulting setup is not really effective. There’s a lot of manual tasks and most of it could be automated in a way to help the researcher.

Spotbugs scans can take multiple hours to complete and use up a lot of resources on large projects. The exhaustion of resources will likely abort the scan and thus waste a lot of time. To prevent this issue, we needed a way to easily off-load the scanning tasks to a more resourceful server. It also needed to be simple to use, having a simple user interface would be ideal.

The Solution

The core of the solution is built around Jenkins CI. We built custom scripts to analyze Java binaries, generate the report and provide a user interface to inspect the code. The choice of Jenkins was quite easy since it has already a large community and a multitude of plugins to add new functionalities.

Some plugins required some adjustments to fit our needs. For instance, the findbugs-plugin was not handling correctly decompiled code. The reported warnings would get highlighted at the beginning of the file or on an unrelated line. With large classes being common in various business softwares, we fixed the issue by publishing a slightly modified version that attempts to identify a relevant location in the class file (such as the beginning of the method). Despite the location in the file being only an approximation, this modification definitely helped reducing the time needed to identify the vulnerable code path.

Static analyzer will often generate lots of false positives. Triaging those findings is, therefore, the main task in vulnerability research. When the code analyzed is fairly small, the Jenkins plugin is fine to do a quick overview of the warnings. But when the number of findings is in the thousands, it is quite a challenge to find and analyze relevant warnings.

In order to provide a single way to access the decompiled code and view the issues in an IDE agnostic manner, each warning is annotated in the code. This annotation takes form of a comment at the end of the line of the relevant method. Locating these comments in a large code base can be rather slow. To overcome this problem, each code annotation is also committed in a Git repository. This way, whether you are using a Git client, an IDE such as IntelliJ IDEA or Emacs, you can inspect the Git commits and go directly to the potentially vulnerable code.

Pipeline

Java binaries are decompiled using CFR. SpotBugs and Find Security Bugs will generate a vulnerability report. This report is available in Jenkins and is also used to annotate the code to produce Git commits.

All those steps are triggered after the upload of a single jar, multiple jars packages inside a zip or an APK (Android).

Our Findings

Over the course of the previous months, we used this tool to scan multiple open source and proprietary software solutions along with several hundred Android applications. We have reported multiple vulnerabilities in critical pieces of software and most of them are trivial to exploit. While we cannot disclose the details of our findings at the moment, we are going to release a blog focused on the findings soon.

In the meantime, the tool suite is available on Github and the only dependency you need to start a scan is docker-compose.

The portable infrastructure is available on GitHub at https://github.com/gosecure/jenkins-fsb.

We believe that by open sourcing this toolchain we will help bug hunters and organizations strengthen the quality of their Java (or JVM) software products or audit the software they purchase resulting in more disclosed bugs and a more resilient ecosystem.

This blog post was written by Benoit Côté-Jodoin as part of an internship with us. We are proud of Benoit’s work and we are glad he accepted the opportunity to share his research and tools with the world.