A few years ago, I worked with a developer who spent an entire Friday afternoon "optimizing" a critical path in our data ingestion pipeline. He had written a small main method, wrapped a loop in System.nanoTime(), and concluded that replacing a List.foldLeft with a while loop had given us a 10x performance boost. He was thrilled. But when we pushed it to production, the throughput barely budged. In some cases, it actually performed worse. The problem wasn't his logic; it was that he was benchmarking the JVM's "cold" state and ignoring the JIT (Just-In-Time) compiler. The JVM had simply optimized away his functional code in the second run because it realized the result wasn't being used, making it look infinitely fast. This is the classic microbenchmark trap.

Why your manual timers are lying to you

Measuring performance on the JVM is notoriously difficult because the environment is dynamic. The JIT compiler monitors your code as it runs; once a method is identified as "hot," the JVM recompiles it into highly optimized machine code. If you just run a piece of code once or twice, you're measuring the interpreter or the initial compilation overhead, not the steady-state performance. Worse yet, if you write a loop that calculates a value but doesn't actually do anything with that value, the compiler might decide the entire operation is "dead code" and remove it entirely. You end up benchmarking a no-op, which is why your results look impossibly fast.

This is where JMH (Java Microbenchmark Harness) comes in. It's the gold standard for JVM benchmarking because it handles the "warm-up" phase for you, ensures the JIT has a chance to optimize, and provides mechanisms to prevent dead-code elimination. Even though it's a Java tool, it's exactly what we use for Scala performance tuning.

Wiring JMH into your Scala project

While you can use the raw JMH library, the most painless way to integrate it into a Scala project is via the sbt-jmh plugin. Once you've added the plugin to your plugins.sbt, you can define your benchmarks in a separate project or a specific directory. I usually recommend a dedicated bench folder to keep the benchmark dependencies from leaking into your production build.

A JMH benchmark is essentially a class where specific methods are marked with the @Benchmark annotation. However, you can't just put your data inside the benchmark method; if you do, the compiler might treat those values as constants and pre-calculate the result at compile time. Instead, we use @State to manage the data we're operating on.

import org.openjdk.jmh.annotations.*
import java.util.concurrent.TimeUnit

@State(Scope.Thread)
@BenchmarkMode(Mode.AverageTime)
@OutputTimeUnit(TimeUnit.NANOSECONDS)
class CollectionBenchmark {
  // We put the data here so the JIT doesn't constant-fold it
  val data = (1 to 1000).toList

  @Benchmark
  def testFoldLeft(): Int = {
    data.foldLeft(0)(_ + _)
  }

  @Benchmark
  def testWhileLoop(): Int = {
    var sum = 0
    val it = data.iterator
    while (it.hasNext) {
      sum += it.next()
    }
    sum
  }
}

Writing benchmarks that the JIT can't cheat

Looking at the example above, notice that both methods return a value. This is critical. If I had changed testFoldLeft to return Unit, the JIT compiler would see that the result of the fold is never used and would likely delete the entire loop. By returning the result, JMH can "consume" it, forcing the JVM to actually execute the instructions.

You'll also notice @State(Scope.Thread). This tells JMH to create one instance of the benchmark class per worker thread. This prevents contention between threads that would skew your results. When you run this via sbt, JMH will run several "warm-up" iterations to let the JVM heat up, followed by several "measurement" iterations. The resulting average is a far more honest representation of how your code will behave in a long-running production service than any System.nanoTime() call could ever provide.

📋 Practical Task

Benchmarking String Concatenation Strategies

In this exercise, you will create a JMH benchmark to compare three different ways of building a string in Scala. This will help you understand when the Scala compiler's built-in optimizations are sufficient and when you need to reach for a StringBuilder.

Requirements:

• Create a benchmark class annotated with @State(Scope.Thread).
• Define a list of 100 small strings (e.g., val fragments = List.fill(100)("scala")) as a member of the state.
• Implement three benchmark methods:
  1. testPlusOperator: Use fragments.foldLeft("")(_ + _) to concatenate the strings.
  2. testStringJoin: Use fragments.mkString("").
  3. testStringBuilder: Manually iterate through the list using a StringBuilder.
• Ensure all methods return the resulting string to prevent dead-code elimination.
• Set the @BenchmarkMode to Mode.AverageTime and @OutputTimeUnit to TimeUnit.MICROSECONDS.

Goal: Run the benchmarks and observe the difference in performance. You should find that the + operator in a fold is significantly slower due to the creation of intermediate string objects, while mkString and StringBuilder perform similarly.