Last Updated on August 26, 2024
Java Microbenchmark Harness (JMH) is an open-source tool developed by the OpenJDK team to build, run, and analyze nano/micro/milli benchmarks written in Java and other languages targeting the JVM.
It addresses the complexities of writing accurate microbenchmarks in Java, considering factors like warm-up, dead-code elimination, and constant folding that can significantly affect results.
Microbenchmarking is notoriously difficult due to various factors like JVM warm-up, JIT compilation, and garbage collection.
JMH provides a framework to write reliable benchmarks that account for these factors.
How it Works?
At its core, JMH operates by creating a controlled environment that minimizes the impact of JVM optimizations and other factors that can skew benchmark results.
When a JMH benchmark is executed, it first compiles the benchmark code along with its own runtime. It then forks a new JVM process for each benchmark, ensuring isolation and preventing interference between tests.
Before taking measurements, JMH runs several warm-up iterations. This crucial step allows the JVM to reach a steady state, applying just-in-time (JIT) compilations and optimizations as it would in a real-world scenario.
JMH begins the actual measurement phase. It executes the benchmark code repeatedly, collecting precise timing data.
To prevent dead code elimination and constant folding—JVM optimizations that could invalidate results—JMH employs techniques like blackhole consumption of results and the use of volatile state objects.
Blackhole consumption in JMH is a technique to prevent the JVM from optimizing away code it thinks is unused.
Volatile state objects in JMH are used to prevent unwanted compiler optimizations. When marked volatile, the JVM cannot cache values or reorder operations on these objects.
Once measurements are complete, JMH performs statistical analysis on the collected data. It calculates metrics such as average time, throughput, and variance, providing a comprehensive view of the code’s performance characteristics.
This methodical approach allows JMH to deliver reliable and reproducible benchmark results, essential for performance tuning and optimization in Java applications.
JMH Configuration
To use JMH, you typically need to set up a Maven project.
Add the JMH dependencies to your pom.xml.
<dependencies>
<dependency>
<groupId>org.openjdk.jmh</groupId>
<artifactId>jmh-core</artifactId>
<version>1.35</version>
</dependency>
<dependency>
<groupId>org.openjdk.jmh</groupId>
<artifactId>jmh-generator-annprocess</artifactId>
<version>1.35</version>
</dependency>
</dependencies>Add the JMH Maven plugin:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<finalName>benchmarks</finalName>
<transformers>
<transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
<mainClass>org.openjdk.jmh.Main</mainClass>
</transformer>
</transformers>
</configuration>
</execution>
</executions>
</plugin>@Benchmark
The @Benchmark annotation is used to mark a method as a benchmark. This indicates that the method should be measured for performance.
@Benchmark
public void myBenchmark() {
// Benchmark code here
}@Setup
The @Setup annotation is used to specify a setup method that will be executed before each benchmark iteration or trial.
This allows you to initialize variables, create objects, or perform other setup tasks that are necessary for the benchmark.
@State(Scope.Thread)
public class MyBenchmarkState {
private List<Integer> data;
@Setup(Level.Trial)
public void setup() {
data = new ArrayList<>();
for (int i = 0; i < 1000; i++) {
data.add(i);
}
}
@Benchmark
public void measureIteration() {
for (int i = 0; i < data.size(); i++) {
// Benchmark code here
}
}
}In this example:
- The
@Setup(Level.Trial)annotation indicates that thesetup()method should be executed before each trial of the benchmark. - The
setup()method initializes thedatalist with random values. - The
measureIteration()benchmark method can access thedatalist from the state object.
@BenchmarkMode
The @BenchmarkMode annotation in JMH specifies how the benchmark results should be interpreted.
Main modes:
- Throughput: Operations per unit of time (e.g., ops/sec)
- AverageTime: Average time per operation
- SampleTime: Samples the time for each operation
- SingleShotTime: Measures the time for a single operation
@State
The @State annotation is used to define a state object that holds the data and setup required for a benchmark. This allows you to create more complex benchmarks that involve multiple setup steps or shared state between different benchmark methods.
@State(Scope.Thread)
public class MyBenchmarkState {
private List<Integer> data;
@Setup(Level.Trial)
public void setup() {
data = new ArrayList<>();
for (int i = 0; i < 1000; i++) {
data.add(i);
}
}
@Benchmark
public void measureIteration() {
for (int i = 0; i < data.size(); i++) {
// Benchmark code here
}
}
}In this example:
- The
@Stateannotation defines aMyBenchmarkStateclass as the state object. - The
@Setupannotation withLevel.Trialspecifies that thesetup()method should be executed before each trial of the benchmark. - The
measureIteration()benchmark method can access thedatafield from the state object.
@Warmup
The @Warmup annotation is used to specify a warmup phase for microbenchmarks. This phase allows the JVM to optimize the code and stabilize performance before the actual measurement begins.
@BenchmarkMode(Mode.AverageTime)
@OutputTimeUnit(TimeUnit.NANOSECONDS)
@Warmup(iterations = 5, time = 1, timeUnit = TimeUnit.SECONDS)
public void myBenchmark() {
// Benchmark code here
}In the above example:
iterations: Specifies the number of warmup iterations.time: Specifies the duration of each warmup iteration.timeUnit: Specifies the time unit for the warmup duration (e.g.,TimeUnit.SECONDS).
@Fork
The @Fork annotation is used to specify the number of forks or separate JVM instances that will be used to run the benchmark.
Each fork represents an independent execution of the benchmark, helping to reduce measurement bias and improve the reliability of the results.
@BenchmarkMode(Mode.AverageTime)
@OutputTimeUnit(TimeUnit.NANOSECONDS)
@Warmup(iterations = 5, time = 1, timeUnit = TimeUnit.SECONDS)
@Fork(value = 3, warmups = 2)
public void myBenchmark() {
// Benchmark code here
}In this example:
warmups: Specifies the number of warmup iterations for each fork.
value: Specifies the number of forks.
@Param
The @Param annotation in JMH (Java Microbenchmark Harness) is used to define parameters for a benchmark. This allows you to easily run the benchmark with different parameter values and compare the results.
@Benchmark
public void myBenchmark(int size) {
@Param({"10", "100", "1000"})
private int size;
}In this example:
@Params({"1000", "10000", "100000"})defines a parameter named “size” with three possible values.
JMH Example Usage
This benchmark compares string concatenation using the ‘+’ operator versus StringBuilder.
import org.openjdk.jmh.annotations.*;
import org.openjdk.jmh.infra.Blackhole;
import java.util.concurrent.TimeUnit;
@BenchmarkMode(Mode.AverageTime)
@OutputTimeUnit(TimeUnit.NANOSECONDS)
@State(Scope.Thread)
@Fork(value = 2, jvmArgs = {"-Xms2G", "-Xmx2G"})
@Warmup(iterations = 3)
@Measurement(iterations = 5)
public class StringBenchmark {
@Param({"10", "100", "1000"})
private int length;
private String data;
@Setup
public void setup() {
data = "";
for (int i = 0; i < length; i++) {
data += "a";
}
}
@Benchmark
public void stringConcat(Blackhole bh) {
String result = "";
for (int i = 0; i < length; i++) {
result += data.charAt(i);
}
bh.consume(result);
}
@Benchmark
public void stringBuilder(Blackhole bh) {
StringBuilder sb = new StringBuilder();
for (int i = 0; i < length; i++) {
sb.append(data.charAt(i));
}
bh.consume(sb.toString());
}
}By running the above program in the console, we consistently observe that string concatenation using the + operator performs significantly worse than using a StringBuilder.
As the length of the strings increases, the performance gap between the two methods widens, emphasizing the importance of using StringBuilder for efficient string concatenation in Java.
Conclusion
Java Microbenchmark Harness (JMH) is a powerful tool for measuring the performance of Java code with precision and reliability.
By providing a controlled environment and advanced features, JMH enables developers to accurately assess the impact of code changes, identify bottlenecks, and optimize their applications.
