I've seen this exact scenario play out in production more times than I care to admit. A developer implements a health check, the Kubernetes pods stay green, and then suddenly, the entire site goes down because the database connection pool exhausted itself, but the load balancer kept sending traffic to the broken nodes anyway.

// The "Naive" Implementation
def healthRoute = HttpRoutes.of[IO] {
  case GET -> Root / "health" => 
    Ok("OK") 
}


The Lying Health Check
At first glance, the code above looks fine. It's fast, it's simple, and it returns a 200 OK. The problem is that this endpoint is lying. It's telling the orchestrator, "I am healthy!" based solely on the fact that the JVM is running and the HTTP server can respond. It isn't actually checking if the application is capable of doing its job.

If your database goes offline or your Kafka producer hangs, this endpoint will still return "OK". Kubernetes will see that 200 response and keep routing user traffic to a pod that is effectively a zombie. Your users get 500 errors, but your monitoring dashboard shows a sea of green. That's a nightmare to debug during an incident.

Splitting Liveness from Readiness
To fix this, we have to stop treating "health" as a single concept. In a professional Scala service, you generally need two distinct endpoints: Liveness and Readiness.

• Liveness: Tells the orchestrator if the app has crashed or deadlocked. If this fails, the pod is killed and restarted. Keep this extremely lightweight.
• Readiness: Tells the orchestrator if the app is ready to handle traffic. If this fails, the pod is removed from the load balancer, but it isn't killed. This is where you check your database, cache, and external APIs.

Implementing a Dependency-Aware Check

Let's rewrite this to actually verify our dependencies. I'll assume you're using a database pool (like HikariCP) and wrapping your logic in cats.effect.IO. We want to attempt a simple query—like SELECT 1—to ensure the connection is actually alive.

case class HealthCheck(dbPool: HikariDataSource) {
  def checkDb: IO[Boolean] = IO.delay {
    try {
      dbPool.getConnection().use { conn =>
        conn.createStatement().executeQuery("SELECT 1").next()
      }
      true
    } catch {
      case _: Exception => false
    }
  }.handleErrorWith(_ => IO.pure(false))
}

def healthRoutes(checker: HealthCheck) = HttpRoutes.of[IO] {
  // Liveness: Just check if the server is up
  case GET -> Root / "live" => 
    Ok("Alive")

  // Readiness: Check if the DB is actually responding
  case GET -> Root / "ready" => 
    checker.checkDb.flatMap {
      case true  => Ok("Ready")
      case false => ServiceUnavailable("Database unreachable")
    }
}


Now, if the database lags or drops, the /ready endpoint starts returning a 503. Kubernetes will stop sending traffic to that specific pod, allowing it to recover or allowing you to see exactly which node is struggling without taking down the entire cluster. I prefer using ServiceUnavailable over a generic 500 because it explicitly tells the infrastructure that the failure is temporary and related to availability, not a code bug.

📋 Practical Task

Implement a Multi-Dependency Readiness Probe

You are managing a service that depends on both a Postgres Database and a Redis Cache. Currently, the service only checks the database.

Your task is to modify the HealthCheck class and the /ready route to implement the following logic:

1. Create a checkRedis method that returns an IO[Boolean] (you can simulate this by calling a mock redisClient.ping()).
2. Update the /ready endpoint so that it only returns Ok("Ready") if both the database and the Redis cache are healthy.
3. If either one fails, return a ServiceUnavailable response that includes a brief message indicating which dependency failed (e.g., "Database unreachable" or "Redis unreachable").

Ensure that a failure in the Redis check does not trigger a pod restart (Liveness), but does prevent the pod from receiving traffic (Readiness).