Suvarna Garge (Editor)

Java Code Coverage Tools

Updated on
Edit
Like
Comment
Share on FacebookTweet on TwitterShare on LinkedInShare on Reddit
Operating system
  
Cross-platform

Website
  
JCov

Type
  
Code coverage

Developer(s)
  
Leonid Arbouzov, Alexander Petrov, Vladimir Generalov, Serguei Chukhontsev, Oleg Uliankin, Gregory Steuck, Pavel Ozhdikhin, Konstantin Bobrovsky, Robert Field, Alexander Kuzmin, Leonid Mesnik, Sergey Borodin, Andrey Titov, Dmitry Fazunenko, Alexey Fedorchenko

Stable release
  
3.0 / September 1, 2014; 2 years ago (2014-09-01)

License
  
GNU Public License(version 2, with the Classpath Exception)

Java Code Coverage Tools are of two types: first, tools that add statements to the source code and require recompilation of the source code. Second, tools that instrument the byte code, either before or during execution. The goal is to find out which parts of the code are tested by registering the lines of code executed when running a test.

Contents

JCov

JCov is the tool which has been developed and used with Sun JDK (and later Oracle JDK) from the very beginning of Java: from the version 1.1. JCov is capable of measuring and reporting Java code coverage. JCov is distributed under the terms of the GNU Public License(version 2, with the Classpath Exception). JCov has been open-sourced as a part of OpenJDK codetools project in 2014. JCov is the only code coverage tool working with a JDK release in development (JDK9 at the time of writing).

Features

JCov is capable of reporting the following types of code coverage:

  • Block coverage
  • Line coverage
  • Branch coverage
  • Method coverage

    • JCov implements two different ways to save the collected data:

  • into a file on the filesystem
  • onto a server (a.k.a. "network grabber").

    • JCov works by instrumenting Java bytecode using two different approaches:

  • static instrumentation which is done upfront, changing the tested code
  • dynamic instrumentation which is done on the fly by means of Java agent.

    • JCov has a few more distinctive features which include, but are not limited to:

  • Field coverage
  • Abstract API coverage
  • Direct/indirect coverage
  • Per-test coverage information (a.k.a. "test scales")
  • Public API and SPI which makes it possible to implement custom filtering and/or mining the coverage data.

    • Tools using JCov

  • Oracle JDK (SE and ME)
  • JCK (the Java Compatibility Kit)
  • Various Java SE and Java ME TCKs
  • Java FX SDK
  • Java FX Scene Builder

    • JaCoCo

    JaCoCo is an open source toolkit for measuring and reporting Java code coverage. JaCoCo is distributed under the terms of the Eclipse Public License. It was developed as a replacement for EMMA under the umbrella of the EclEmma plug-in for Eclipse.

    Features

    JaCoCo offers instructions, line and branch coverage.

    In contrast to Clover, which requires instrumenting the source code, JaCoCo can instrument Java bytecode using two different approaches:

  • like JCov on the fly while running the code with a Java agent,
  • like JCov and Cobertura prior to execution (offline)

    • And can be configured to store the collected data in a file, or send it via TCP. Files from multiple runs or code parts can be merged easily. Unlike Cobertura and Emma it fully supports Java 7 and Java 8.

    Tools using or including JaCoCo

  • SonarQube JaCoCo plugin — one of the defaults for coverage analyses within the code quality management platform SonarQube
  • EclEmma Eclipse (software) Code Coverage Plugin, was formerly EMMA based
  • Jenkins JaCoCo Plugin
  • Netbeans JaCoCo support
  • IntelliJ IDEA since v11
  • Gradle JaCoCo Plugin

    • Clover

    Clover is a Java Code Coverage Analysis application bought and further developed by Atlassian. It is a commercial product freely available to open source projects and non-profit institutions.

    Clover is using a source code instrumentation technique (as opposed to Cobertura and JaCoCo which are using byte code instrumentation), which has its advantages (such as an ability to collect code metrics) and disadvantages (re-compilation of sources is necessary). Some of its features include historical reporting, huge control over the coverage gathering process, command line toolset and API for legacy integration and more.

    Clover also allows testing time to be reduced by only running the tests that cover the application code that was modified since the previous build. This is called Test Optimization and can lead to huge drops in the amount of time spent waiting for automated tests to complete.

    Clover comes with a number of integrations both developed by Atlassian (Ant, Maven, Grails, Eclipse, IDEA, Bamboo) and by open source community (Gradle, Griffon, Jenkins, Hudson, Sonar).

    Cobertura

    Cobertura is an open source tool for measuring code coverage. It does so by instrumenting the byte code.

    EMMA

    EMMA is an open source toolkit for measuring and reporting Java code coverage. EMMA is distributed under the terms of Common Public License v1.0.

    EMMA is not currently under active development; the last stable release took place in mid-2005. As replacement, JaCoCo was developed. EMMA works by wrapping each line of code and each condition with a flag, which is set when that line is executed.

    Features

  • instrument classes for coverage either offline (before they are loaded) or on the fly (using an instrumenting application classloader).
  • Supported coverage types: class, method, line, basic block. EMMA can detect when a single source code line is covered only partially.
  • Coverage stats are aggregated at method, class, package, and "all classes" levels.
  • Output report types: plain text, HTML, XML. All report types support drill-down, to a user-controlled detail depth. The HTML report supports source code linking.
  • Output reports can highlight items with coverage levels below user-provided thresholds.
  • Coverage data obtained in different instrumentation or test runs can be merged.
  • it is possible to dump or reset coverage data remotely and without a JVM exit.
  • does not require access to the source code and degrades gracefully with decreasing amount of debug information available in the input classes.
  • can instrument individual .class files or entire .jars (in place, if desired). Efficient coverage subset filtering is possible, too.
  • Makefile and ANT build integration are supported on equal footing.
  • The runtime overhead of added instrumentation is small (5–20%) and the bytecode instrumentor itself is very fast (mostly limited by file I/O speed). Memory overhead is a few hundred bytes per Java class.
  • EMMA is 100% pure Java, has no external library dependencies, and works in any Java 2 JVM (even 1.2.x).

    • EMMA-based tools

  • Intellij Idea Plugin—Includes code coverage support from its own and an EMMA plugin
  • SonarQube EMMA plugin—Enables usage of EMMA analyses within the code quality management platform SonarQube
  • Google CodePro AnalytiX
  • Jenkins Emma Plugin

    • Serenity

    Serenity is an open source toolkit for measuring and reporting Java code coverage. As well as coverage, major code metrics are measured:- cyclometric complexity, stability, abstractness and distance from main. The report data is persisted to an object database, and made available via Jenkins/Hudson. The interface replicates the Eclipse IDE interface visually.

    Serenity dynamically enhances the byte code, making a post-compile step unnecessary. Ant and Maven projects are supported. Configuration is done in xml, an Ant example would be:

    And a Maven configuration example would be:

    For a full example of a configuration please refer to the Jenkins wiki at https://wiki.jenkins-ci.org/display/JENKINS/Serenity+Plugin.

    Jenkins slaves as well as Maven multi module projects are supported.

    Testwell CTC++ for Java

    Testwell CTC++ is a Code Coverage Analyser for C, C++, Java and C#. The development of this tool started in 1989 at Testwell in Finland. Since 2013 support and development has been continued by Verifysoft Technology, a company from Offenburg, Germany. Testwell CTC++ analyses for all code coverage levels up to Modified condition/decision coverage and Multicondition Coverage. The tool works with all compilers.

    References

    Java Code Coverage Tools Wikipedia
    (Text) CC BY-SA