I once spent an entire Tuesday morning writing a complex regular expression to parse through five gigabytes of production logs. I was trying to find every instance where a specific set of payment parameters caused a timeout, but because the original developer used string interpolation for everything, the logs looked like this: Processed payment for user 123 with amount 50.00 in 200ms. Simple, right? Until another developer changed it to Payment processed: user=123, amt=50.00, time=200ms. My regex broke, and I wasted hours chasing a ghost.

case class PaymentRequest(userId: String, amount: Double, currency: String)

class PaymentService {
  val logger = org.slf4j.LoggerFactory.getLogger(getClass)

  def processPayment(req: PaymentRequest): Unit = {
    // This looks fine during local development...
    logger.info(s"Processing payment for user ${req.userId} of ${req.amount} ${req.currency}")
    
    try {
      // simulate logic
    } catch {
      case e: Exception => 
        logger.error(s"Payment failed for user ${req.userId}: ${e.getMessage}")
    }
  }
}

The Nightmare of String Interpolation

The code above is what I call "human-readable, machine-useless." When you're staring at a terminal, Processing payment for user 123... is great. But when you move to a log aggregator like ELK (Elasticsearch, Logstash, Kibana) or Datadog, you don't want to "read" logs; you want to query them. If you want to find the average payment amount for users in the last hour, you can't do that with a string. You'd have to pull every single log line and run a regex parser over them in your head or via a slow script.

Turning Logs into Searchable Data

Structured logging flips the script. Instead of baking the data into a message string, we treat the log message as a static identifier and the data as a set of key-value pairs. In the Scala ecosystem, we usually achieve this by using a library like logback-scala or log4cats, paired with a JSON encoder (like the Logstash Logback Encoder).

Here is how I would rewrite that service to actually be useful in a production environment:

import net.logstash.logback.argument.StructuredArguments._

class PaymentService {
  val logger = org.slf4j.LoggerFactory.getLogger(getClass)

  def processPayment(req: PaymentRequest): Unit = {
    // We use a static message and pass the data as structured arguments
    logger.info("Processing payment", 
      kv("userId", req.userId), 
      kv("amount", req.amount), 
      kv("currency", req.currency)
    )
    
    try {
      // simulate logic
    } catch {
      case e: Exception => 
        logger.error("Payment failed", 
          kv("userId", req.userId), 
          kv("error", e.getMessage),
          kv("stacktrace", e.getStackTrace)
        )
    }
  }
}

Why This Actually Works

When this code runs with a JSON appender, it doesn't produce a sentence. It produces a JSON object:

{
  "@timestamp": "2023-10-27T10:00:00.000Z",
  "message": "Processing payment",
  "userId": "123",
  "amount": 50.00,
  "currency": "USD",
  "level": "INFO"
}


Now, if I need to find all payments over $1,000, I don't write a regex. I write a query: amount > 1000. It's instantaneous, it's precise, and it doesn't break when someone changes the word "Processing" to "Handling".

One thing to watch out for: don't go overboard. I've seen developers log entire 50-field domain objects into every single log line. This bloats your storage costs and can actually slow down your application because of the serialization overhead. Only log the IDs and the specific fields you actually intend to filter or aggregate by.

📋 Practical Task

Refactoring the OrderProcessing Pipeline for JSON Log Observability

You have been handed a legacy OrderProcessor class that uses standard string interpolation for its logging. This has made it impossible for the SRE team to track order failure rates by region.

Your Task: Refactor the processOrder method to use structured logging. Replace the string interpolation in both the success and error paths with kv("key", value) pairs using the net.logstash.logback.argument.StructuredArguments library.

case class Order(id: String, customerId: String, total: Double, region: String)

class OrderProcessor {
  val logger = org.slf4j.LoggerFactory.getLogger(getClass)

  def processOrder(order: Order): Unit = {
    // TODO: Refactor this log to be structured
    logger.info(s"Starting processing for order ${order.id} in region ${order.region}")

    try {
      if (order.total < 0) throw new IllegalArgumentException("Negative total")
      
      // TODO: Refactor this log to be structured
      logger.info(s"Successfully processed order ${order.id} for customer ${order.customerId}")
    } catch {
      case e: Exception => 
        // TODO: Refactor this log to be structured
        logger.error(s"Order ${order.id} failed: ${e.getMessage}")
    }
  }
}

Requirements:

• The "Starting" log must include orderId and region.
• The "Success" log must include orderId and customerId.
• The "Failure" log must include orderId and the error message.
• All log messages must be converted to static strings (no s"...").