Skip to main content

← All tools & datasets

SpotJMHBugs

SpotJMHBugs is a SpotBugs plugin that statically analyzes JMH microbenchmarks and flags bad practices that can silently invalidate benchmark results.

Source code: github.com/DiegoEliasCosta/spotjmhbugs (MIT license)

Bad practices detected​

CodeBad practice
RETUNot using the returned computation
LOOPUsing accumulation to consume values inside a loop
FINALUsing final primitives for benchmark inputs
INVORunning fixture methods for each benchmark method invocation
FORKConfiguring benchmarks with zero forks

Build​

Requires Java 8 or higher.

git clone https://github.com/DiegoEliasCosta/spotjmhbugs.git
cd spotjmhbugs
./mvnw package   # produces spotJMHbugs.jar

Use it with the SpotBugs command line​

  1. Download a SpotBugs binary release from github.com/spotbugs/spotbugs/releases and set SPOTBUGS_HOME to the extracted folder.
  2. Copy spotJMHbugs.jar into $SPOTBUGS_HOME/plugin/.
  3. Check that the plugin is loaded:
java -jar $SPOTBUGS_HOME/lib/spotbugs.jar -textui -showPlugins

Use it in a Maven project​

Add the spotbugs-maven-plugin with SpotJMHBugs as a plugin dependency, and exclude JMH-generated classes with a spotbugs-exclude.xml filter. Running mvn verify then fails the build when bad practices are found. See the repository README for the full pom.xml and filter configuration.

How to cite​

Diego Costa, Cor-Paul Bezemer, Philipp Leitner, and Artur Andrzejak. What's Wrong With My Benchmark Results? Studying Bad Practices in JMH Benchmarks. IEEE Transactions on Software Engineering (TSE), 2019.