When we talk about "observability," it's easy to get lost in the marketing jargon of cloud providers. But for us as engineers, it really boils down to one question: "What is the code actually doing right now?" Log files are great for post-mortems, but they're terrible for seeing trends. That's why I use Kamon. It gives us those high-level metrics—counters, timers, and gauges—without forcing us to rewrite our entire architecture.

To show you how this works, we're going to build a simple OrderProcessor. It's a dummy service that simulates processing a customer order, occasionally failing, and occasionally taking a long time to respond. We want to track how many orders we've processed and the latency of those operations.

Defining our business metrics

First, I'll set up the core logic. I want to know two things: how many orders were successful and how many failed. In Kamon, we use counters for this. I'm going to define these at the object level so they persist across the lifecycle of the application.

import kamon.Kamon
import kamon.metric.Counter
import scala.util.{Random, Try, Success, Failure}

object OrderProcessor {
  // We define these once. Kamon handles the thread-safe incrementing.
  val successCounter: Counter = Kamon.defineCounter("order.processed.success")
  val failureCounter: Counter = Kamon.defineCounter("order.processed.failure")

  def process(orderId: String): Unit = {
    println(s"Processing order $orderId...")
    
    // Simulate some random failure
    if (Random.nextInt(10) > 7) {
      failureCounter.increment(1)
      throw new RuntimeException("Database timeout!")
    }
    
    successCounter.increment(1)
    println(s"Order $orderId completed.")
  }
}

Measuring latency with timers

Counting successes is fine, but it doesn't tell me if the system is slowing down. I need a Timer. Instead of manually starting and stopping a stopwatch, Kamon lets us wrap a block of code. I'll use Kamon.defineTimer and the measure method. This is much cleaner than doing the math myself.

import kamon.metric.Timer

object OrderProcessor {
  val successCounter: Counter = Kamon.defineCounter("order.processed.success")
  val failureCounter: Counter = Kamon.defineCounter("order.processed.failure")
  
  // A timer tracks the duration of the wrapped block
  val processingTimer: Timer = Kamon.defineTimer("order.processing.time")

  def process(orderId: String): Unit = {
    processingTimer.measure {
      println(s"Processing order $orderId...")
      
      // Simulate variable latency
      Thread.sleep(Random.nextInt(200)) 

      if (Random.nextInt(10) > 7) {
        failureCounter.increment(1)
        throw new RuntimeException("Database timeout!")
      }
      
      successCounter.increment(1)
      println(s"Order $orderId completed.")
    }
  }
}

Fixing the "Empty Metrics" ghost

Now, here is where I usually trip up the first time I set this up in a new project. I ran this code, looped through a few hundred orders, and then checked my Prometheus endpoint only to find... absolutely nothing. No counters, no timers, just a blank page.

I spent ten minutes checking my code, thinking I'd messed up the measure block. But the problem wasn't in the Scala code; it was in the configuration. Kamon is designed to be "pluggable." I had the kamon-core and kamon-prometheus dependencies in my build.sbt, but I forgot to actually tell Kamon to start the Prometheus reporter in my application.conf.

If you don't explicitly enable the reporter, Kamon collects the data in memory but doesn't send it anywhere. I had to add this to my config:

# application.conf
kamon.prometheus.reporter {
  # This starts a tiny HTTP server on port 9000 to expose metrics
  port = 9000
}

Once I added that and restarted, the metrics suddenly appeared. It's a classic "silent failure" that catches almost everyone.

Wiring it all together

To see it in action, let's run a quick loop. I'll wrap the call in a Try so the app doesn't crash when our simulated "Database timeout" hits.

object Main extends App {
  (1 to 100).foreach { i =>
    Try(OrderProcessor.process(s"ORD-$i")) match {
      case Success(_) => // All good
      case Failure(e) => println(s"Order $i failed: ${e.getMessage}")
    }
  }
  
  println("Metrics are now available at http://localhost:9000/metrics")
  // Keep the app alive so we can scrape the endpoint
  Thread.sleep(100000)
}

When you hit that URL, you'll see lines like order_processed_success_count 82.0 and order_processing_time_sum. You can now see exactly how your system is behaving without digging through megabytes of logs.

📋 Practical Task

Implementing a Latency Alert Gauge for the OrderProcessor

In the previous example, we tracked the average time and total counts. However, we need to know the "Current Queue Depth"—essentially, how many orders are being processed simultaneously at any given moment.

Your task is to modify the OrderProcessor to implement a Gauge. Unlike a Counter, a Gauge can go up and down.

• Define a new Gauge named "order.concurrent.processing".
• Update the process method so that it increments the Gauge at the very start of the method and decrements it in a finally block to ensure the count is always accurate, even if an exception is thrown.
• Verify that if you simulate multiple threads calling process, the Gauge reflects the number of active threads.