Benchmarking Java

Naive System.currentTimeMillis() timing routinely lies about Java performance — proper benchmarking accounts for JIT warm-up and compiler tricks that would otherwise erase the very code being measured.

Benchmarking is the disciplined measurement of how fast code runs, under conditions realistic enough that the number means something. Naive hand-rolled timing in Java is notoriously misleading because of how the JVM actually executes code: interpretation followed by just-in-time (JIT) compilation, and aggressive compiler optimizations that can eliminate the very code a benchmark intends to measure. This topic covers why naive timing fails, what JIT warm-up means for measurement, dead-code elimination and constant folding as benchmarking hazards, and JMH (Java Microbenchmark Harness) as the standard tool that addresses these problems.

Why Naive Timing Is Misleading

A tempting but flawed approach to measuring performance is to record System.nanoTime() before and after a block of code and compute the difference. On a real JVM this number is unreliable for several compounding reasons: the code may not yet be JIT-compiled (so early iterations run in the slower interpreter), the JVM's optimizer may notice the benchmarked computation's result is never used and eliminate it entirely, the operating system scheduler and garbage collector introduce noise, and a single run captures one sample rather than a statistical distribution.

This measures a mix of interpreted execution, JIT compilation happening mid-loop, and possibly a computation the compiler can prove is unused and skip — none of which reflects steady-state performance.

// Naive, unreliable benchmark — do not do this for real measurements
long start = System.nanoTime();
for (int i = 0; i < 1_000_000; i++) {
    compute(i);
}
long elapsed = System.nanoTime() - start;
System.out.println("Elapsed: " + elapsed + " ns");

JIT Compilation and Warm-Up

The JVM initially interprets bytecode. As a method is invoked repeatedly, the JIT compiler identifies 'hot' methods and compiles them to optimized native machine code, often in multiple tiers of increasing optimization (a typical HotSpot JVM uses tiered compilation, moving from a fast, simply-optimized compile to a more aggressively optimized one as a method proves itself hot). This means the same code can run measurably slower during its first thousands of invocations than after the JIT has fully optimized it.

'Warm-up' means running the code enough times before measuring that the JIT has stabilized its compilation of the hot paths, so the timed portion reflects steady-state performance rather than a mix of interpretation and mid-flight compilation. Skipping warm-up is one of the most common reasons naive benchmarks produce numbers that don't match real production behavior.

Exactly how many iterations constitute 'warm' is JVM-implementation- and code-path-dependent — there is no universal fixed number. This is precisely the kind of concern a proper benchmarking harness automates rather than leaving to guesswork.

Dead-Code Elimination

A sufficiently smart JIT compiler can prove that a computed value is never used (its result is never read, stored, or has any observable effect) and eliminate the computation entirely — this is dead-code elimination, ordinarily a good optimization, but a benchmarking hazard when it silently removes the exact code a benchmark meant to time.

If compute() has no observable side effects and its result is discarded, the JIT is free to optimize the call away — the benchmark would then measure an empty loop, not compute()'s actual cost.

The standard defense is to consume every computed result somehow — accumulate it into a value that is later used or returned, so the compiler cannot prove it is safe to discard. JMH provides a dedicated Blackhole utility for exactly this purpose.

// The result of compute(i) is never used anywhere -
// a sufficiently aggressive optimizer may eliminate the call entirely,
// making the loop measure close to nothing.
for (int i = 0; i < 1_000_000; i++) {
    compute(i); // return value discarded
}

Constant Folding

Constant folding is when the compiler computes a result from known-constant inputs at compile time (or during JIT optimization) instead of at run time. If a benchmark's inputs are effectively constant — the same literal value used every iteration — the compiler may fold the entire computation down to its final answer, again meaning the benchmark times almost nothing.

Using varying, unpredictable-to-the-compiler input (e.g. values from a benchmark parameter, or Blackhole-consumed input) prevents the compiler from proving the computation is constant and folding it away.

// If x is always 5 and never varies across iterations, the JIT may
// fold the arithmetic into a constant, defeating the benchmark's intent.
int x = 5;
for (int i = 0; i < 1_000_000; i++) {
    int result = x * x + 2 * x + 1; // may be folded to 36 once, if provably constant
}

JMH: Java Microbenchmark Harness

JMH (developed by the OpenJDK team specifically to build correct microbenchmarks) is the standard tool for benchmarking Java code. It runs code in separate forked JVM processes, automates warm-up iterations before measurement iterations, runs multiple measurement iterations to gather a statistical distribution, and provides a Blackhole utility to consume results and prevent dead-code elimination — directly addressing the pitfalls above rather than leaving them to the benchmark author to remember.

@Warmup and @Measurement iterations are configured explicitly and separately, @Fork(1) runs the benchmark in a fresh JVM process to avoid cross-contamination between benchmarks, and returning the result from each @Benchmark method (rather than discarding it) avoids dead-code elimination without needing an explicit Blackhole here.

JMH is typically run as a separate build (a dedicated Maven/Gradle module with the JMH annotation processor), producing a runnable benchmark JAR — it is not simply a library called from application code, because it needs its own build-time code generation and process-forking control.

import org.openjdk.jmh.annotations.*;
import org.openjdk.jmh.infra.Blackhole;
import java.util.concurrent.TimeUnit;

@BenchmarkMode(Mode.AverageTime)
@OutputTimeUnit(TimeUnit.NANOSECONDS)
@Warmup(iterations = 5, time = 1, timeUnit = TimeUnit.SECONDS)
@Measurement(iterations = 5, time = 1, timeUnit = TimeUnit.SECONDS)
@Fork(1)
@State(Scope.Thread)
public class StringConcatBenchmark {

    private String a = "hello";
    private String b = "world";

    @Benchmark
    public String concatWithPlus() {
        return a + b; // consumed via method return, so the JIT cannot discard it
    }

    @Benchmark
    public String concatWithBuilder() {
        return new StringBuilder(a).append(b).toString();
    }
}

Statistically Meaningful Measurements

A single elapsed-time number tells you almost nothing about a system with any variance (and virtually every real system has variance from GC, OS scheduling, thermal throttling, and JIT recompilation). A meaningful benchmark result reports a distribution: mean, standard deviation or a confidence interval, across many independent measurement iterations, ideally across multiple JVM forks so no single benchmark's JIT compilation state contaminates another's.

JMH's default output includes the mean along with an error margin (a confidence interval), and supports multiple benchmark modes: average time per operation, throughput (operations per unit time), sample time (percentile distribution of individual invocation times), and single-shot time (for measuring cold-start cost specifically).

Comparing Alternatives Fairly

When benchmarking to compare two approaches (e.g. StringBuilder vs. string concatenation, or two collection implementations), every confounding variable should be controlled: same JVM version and flags, same input data and size, same warm-up regimen, same machine under similar load, and ideally alternating which benchmark runs first across forks to cancel out any systematic bias (e.g. thermal state, background OS activity) that might favor whichever ran first or second.

When Micro-benchmarks Mislead

A microbenchmark measures one isolated piece of code under idealized, repeated-invocation conditions — this can differ substantially from how that code behaves inside a full application, where it runs alongside many other hot methods competing for JIT compilation resources, cache lines, and CPU cores, and where it may run cold (called only occasionally) rather than millions of times in a tight loop. A microbenchmark result is a useful data point, not a guarantee of identical real-world impact — it should be complemented by profiling the actual application (see the Profiling topic) when the stakes are high.

Benchmark Modes and Choosing What to Report

JMH supports several distinct benchmark modes because 'how fast' means different things depending on the question being asked. Average time reports mean time per operation, useful for typical latency questions. Throughput reports operations completed per unit time, useful for capacity-planning questions. Sample time records a distribution of individual invocation times and reports percentiles (p50, p99, etc.), useful when tail latency matters more than the average. Single-shot time measures a single cold invocation with no warm-up, useful for estimating cold-start or one-off cost rather than steady-state performance.

Reporting only a mean can hide a problem that a p99 or a throughput number would reveal (e.g. an approach with a lower average time but occasional long pauses from allocation-triggered GC) — choose the mode that matches what the benchmark result will actually be used to decide.

Key Takeaways

Hand-rolled System.nanoTime() timing is unreliable because of JIT warm-up effects, dead-code elimination, constant folding, and single-sample noise.

JIT warm-up means code runs measurably slower before the compiler has optimized its hot paths — benchmarks must discard early, unwarmed iterations.

Dead-code elimination can silently remove an unused computed result; always consume benchmark results (return them, or use a Blackhole).

Constant folding can collapse a computation over unvarying inputs into a precomputed constant, again measuring nothing meaningful.

JMH automates forking, warm-up, measurement iterations, and result consumption, which is why it is the standard tool for Java microbenchmarks rather than hand-written timing loops.

Report a distribution (mean plus error margin across iterations/forks), not a single elapsed-time sample.

A microbenchmark result describes isolated behavior; validate important conclusions against real application profiling too.

Name three distinct reasons a hand-rolled System.nanoTime() loop can produce a misleading benchmark number.

JIT warm-up effects (early iterations run interpreted or under a less-optimized compile tier), dead-code elimination (the compiler discards a computation whose result is never used), and constant folding (the compiler precomputes a result from unvarying inputs) — plus general single-sample noise from GC and OS scheduling.

What does 'warming up' a benchmark mean, and why does it matter?

It means running the code enough times before measurement that the JIT compiler has finished optimizing its hot paths, so the measured portion reflects steady-state compiled performance rather than a mix of interpretation and mid-flight compilation, which would otherwise inflate or distort the result.

How does dead-code elimination threaten the validity of a benchmark, and how is it typically avoided?

If a computed value is never used, the JIT may prove it has no observable effect and remove the computation entirely, making the benchmark measure close to nothing. It's avoided by consuming every result — returning it from the benchmark method or passing it to a tool like JMH's Blackhole.

Why does JMH fork a separate JVM process for benchmarks rather than running them all in one process?

To prevent one benchmark's JIT compilation state, class loading, and warmed-up code from contaminating the measurement of a different benchmark run afterward in the same process — each fork starts JIT compilation fresh.

What is constant folding, and how can it invalidate a benchmark?

It's a compiler optimization that precomputes a result from inputs known to be constant. If a benchmark repeatedly computes over the same literal/unvarying input, the compiler may fold the whole computation into one precomputed value, so the loop ends up measuring almost nothing.

Why is a single elapsed-time measurement insufficient for a trustworthy benchmark result?

Real systems have variance (GC pauses, OS scheduling, thermal effects, JIT state), so one sample can be an outlier in either direction. A trustworthy result reports a distribution — mean with an error margin — across many measurement iterations and ideally multiple JVM forks.

Why might a JMH microbenchmark result not fully predict a method's performance inside a large real application?

A microbenchmark runs the code in isolation under idealized, repeated-invocation conditions; inside a real application the same code competes with many other hot methods for JIT compilation resources and CPU cache, and may run far less frequently — so results should be treated as a data point, corroborated with real application profiling when stakes are high.

What is wrong with this benchmark attempt, and how would you fix it using proper benchmarking principles (described conceptually, not necessarily full JMH code)?

This runs fibonacci(30) exactly once, with no warm-up, so it measures mostly interpreted execution rather than JIT-compiled steady-state performance, and it's a single sample with no statistical confidence. A proper fix: run many warm-up iterations first (discarding their timings), then run many measurement iterations and report a mean and error margin — which is exactly what JMH's @Warmup and @Measurement annotations automate.

Write a minimal JMH benchmark class comparing ArrayList.add versus LinkedList.add for appending 1000 integers, following correct JMH conventions (state object, warm-up/measurement annotations).

import org.openjdk.jmh.annotations.*; import java.util.*; import java.util.concurrent.TimeUnit; @BenchmarkMode(Mode.AverageTime) @OutputTimeUnit(TimeUnit.MICROSECONDS) @Warmup(iterations = 3) @Measurement(iterations = 5) @Fork(1) @State(Scope.Thread) public class ListAppendBenchmark { @Benchmark public List<Integer> arrayListAppend() { List<Integer> list = new ArrayList<>(); for (int i = 0; i < 1000; i++) list.add(i); return list; // returned so JIT cannot eliminate the work } @Benchmark public List<Integer> linkedListAppend() { List<Integer> list = new LinkedList<>(); for (int i = 0; i < 1000; i++) list.add(i); return list; } }

Each @Benchmark method builds and returns the populated list so the compiler cannot discard the work as dead code, @Warmup/@Measurement separate JIT stabilization from actual measurement, and @Fork(1) isolates the two benchmarks' JIT state from each other. Running this typically shows ArrayList.add appending at the end is fast (amortized O(1)), and for pure sequential append LinkedList is also competitive — the interesting difference between them shows up for random-access or mid-list insertion patterns instead.

A colleague benchmarks string concatenation with for (int i = 0; i < 100; i++) { s = s + i; } inside a JMH @Benchmark method but never returns or otherwise uses s. What benchmarking hazard does this risk, and how should it be fixed?

Because the final value of s is never consumed, an aggressive JIT could, in principle, determine the loop's result is unused and eliminate some or all of the computation (dead-code elimination), understating the real cost. The fix is to return s from the benchmark method (or pass it to a JMH Blackhole.consume() call) so the compiler must treat it as a used, observable value.

long start = System.nanoTime();
int result = fibonacci(30);
long elapsed = System.nanoTime() - start;
System.out.println("Took " + elapsed + " ns, result=" + result);