I've seen it happen a dozen times: a developer wants to optimize a hot path in their code, so they wrap a function call in measureTimeMillis { ... }, run it once, and declare a winner. It feels intuitive. If the stopwatch says 10ms for Method A and 15ms for Method B, Method A is faster, right? In the world of the JVM, that assumption is a dangerous trap.

The temptation of the simple stopwatch

When you're just starting to care about performance, the "naive" approach is usually to write a main function that iterates a few thousand times and prints the elapsed time. Let's say we're trying to figure out the fastest way to concatenate a list of 100 strings into a single comma-separated value. You might write something like this:

fun naiveBenchmark() {
    val data = List(100) { "Item $it" }
    val start = System.currentTimeMillis()
    for (i in 1..1000) {
        var result = ""
        for (item in data) {
            result += "$item,"
        }
    }
    println("Took ${System.currentTimeMillis() - start} ms")
}

On the surface, this looks fine. But you're not actually measuring your code; you're measuring the JVM's attempt to figure out what you're doing. The JVM is an adaptive machine. When it sees a piece of code running repeatedly, the Just-In-Time (JIT) compiler kicks in and optimizes it. Your first few hundred iterations are slow because they're interpreted; the next few thousand are fast because they've been compiled to native machine code. By using a simple stopwatch, you're blending these two entirely different performance profiles into one misleading average.

Fighting the JIT and the Optimizer

There's a more sinister problem here: Dead Code Elimination (DCE). Notice in my example above that the result variable is never used after the loop. A smart compiler—and the JVM is very smart—will realize that the entire inner loop has no side effects and doesn't contribute to the program's output. It might literally delete your code from the execution path. You'll see a benchmark result of "0 ms" and think you've discovered a magical, infinite-speed algorithm, when in reality, you've just measured the speed of doing absolutely nothing.

To get a real number, you'd have to manually "warm up" the JVM by running the code thousands of times before starting the timer, and you'd have to print the result to the console to prevent the optimizer from deleting it. Doing this manually is tedious and error-prone. This is why we use JMH (Java Microbenchmark Harness).

Letting JMH handle the heavy lifting

JMH isn't just a library; it's a framework designed specifically to thwart the JVM's attempts to "cheat" during a benchmark. Instead of a main function, you define benchmarks using annotations. I prefer JMH because it separates the warmup phase from the measurement phase. It runs your code enough times to trigger the JIT compiler so that by the time it starts recording data, the code is running in its "steady state."

Here is how I would approach that same string concatenation problem using JMH:

@State(Scope.Thread)
@BenchmarkMode(Mode.AverageTime)
@OutputTimeUnit(TimeUnit.NANOSECONDS)
open class StringBenchmark {
    private val data = List(100) { "Item $it" }

    @Benchmark
    fun testPlusOperator(blackhole: Blackhole) {
        var result = ""
        for (item in data) {
            result += "$item,"
        }
        blackhole.consume(result)
    }

    @Benchmark
    fun testStringBuilder(blackhole: Blackhole) {
        val sb = StringBuilder()
        for (item in data) {
            sb.append(item).append(",")
        }
        blackhole.consume(sb.toString())
    }
}

Notice the Blackhole parameter. This is the "secret sauce." Instead of printing to the console (which is slow and would ruin your numbers), you pass the result to the Blackhole. This tells the JVM, "I'm using this value," which prevents the compiler from optimizing the code away. By using @BenchmarkMode(Mode.AverageTime), JMH will run the method across multiple iterations and provide a statistically sound average, along with a margin of error.

The trade-off is that JMH requires more boilerplate and a specific project setup (usually via a Gradle plugin), but the result is a measurement you can actually trust. When you're making decisions that affect the latency of a production system, "roughly 10ms" isn't good enough—you need the precision that only a harness like JMH can provide.

📋 Practical Task

Benchmarking List Filtering vs. Sequence Filtering

Create a JMH benchmark class to compare the performance of filtering a large list of integers using a standard List (eager evaluation) versus a Sequence (lazy evaluation).

• Create a list of 10,000 random integers as part of your @State.
• Implement one benchmark method that uses .filter { ... }.map { ... }.toList() on the list.
• Implement a second benchmark method that uses .asSequence().filter { ... }.map { ... }.toList().
• Use a Blackhole to consume the final list of both methods.
• Configure the benchmark to use Mode.AverageTime and TimeUnit.MICROSECONDS.

Run the benchmark and observe whether the overhead of creating a Sequence outweighs the benefit of lazy evaluation for a dataset of this size.