I've seen this happen in more than one production outage: a developer creates a /health endpoint that simply returns http.StatusOK and a "OK" string. They tell the SRE team, "The health check is implemented," and everyone breathes a sigh of relief. Then, the database connection pool exhausts or a critical Redis cache goes down. The load balancer keeps sending traffic to the instance because the health check is still returning 200 OK—even though the service is effectively a zombie, unable to process a single real request.

The "If it responds, it's healthy" fallacy

The misconception is that a responding HTTP server equals a functioning application. In Go, your http.ListenAndServe call might be humming along perfectly, but that only proves the Go runtime is alive and the TCP socket is open. It tells you nothing about the dependencies your code needs to actually do its job.

If your service is a middleware layer that fetches data from a Postgres DB and pushes it to a Kafka topic, a health check that doesn't verify those connections is lying to your orchestrator (like Kubernetes). You'll end up in a "black hole" scenario where the load balancer thinks the node is healthy, but every single user request is returning a 500 Internal Server Error.

Distinguishing Liveness from Readiness

To fix this, you need to stop thinking about "health" as a single binary state. In professional Go services, we usually split this into two distinct concerns: Liveness and Readiness.

• Liveness: "Am I crashed?" This is the shallow check. If this fails, the orchestrator kills the pod and restarts it. A simple 200 OK is actually appropriate here.
• Readiness: "Am I ready to handle traffic?" This is the deep check. If this fails, the load balancer stops sending traffic to this instance, but it doesn't necessarily kill the process. This is where you check your DB, your caches, and your required downstream APIs.

Here is how I typically implement a readiness check for a service that depends on a database. I don't just check if the DB variable is non-nil; I actually ping the database to ensure the connection is still viable.

package main

import (
	"database/sql"
	"net/http"
	"time"

	_ "github.com/lib/pq"
)

type HealthHandler struct {
	db *sql.DB
}

func (h *HealthHandler) ReadinessHandler(w http.ResponseWriter, r *http.Request) {
	// We use a timeout context here because we don't want the health check 
	// itself to hang the server if the DB is completely unresponsive.
	ctx, cancel := context.WithTimeout(r.Context(), 2*time.Second)
	defer cancel()

	if err := h.db.PingContext(ctx); err != nil {
		// Log the error internally, but return a 503 Service Unavailable
		// so the load balancer knows to pull us out of rotation.
		http.Error(w, "Service Unavailable: Database unreachable", http.StatusServiceUnavailable)
		return
	}

	w.WriteHeader(http.StatusOK)
	w.Write([]byte("Ready"))
}

func main() {
	db, _ := sql.Open("postgres", "connection_string")
	h := &HealthHandler{db: db}

	mux := http.NewServeMux()
	// Liveness: just proves the app is running
	mux.HandleFunc("/livez", func(w http.ResponseWriter, r *http.Request) {
		w.WriteHeader(http.StatusOK)
	})
	// Readiness: proves the app can actually work
	mux.HandleFunc("/readyz", h.ReadinessHandler)

	http.ListenAndServe(":8080", mux)
}

Notice a few key details here. First, I used PingContext with a strict timeout. You never want your health check to be the reason your app freezes. Second, I'm returning a 503 Service Unavailable rather than a 500. In the world of infrastructure, a 503 specifically signals that the server is currently unable to handle the request due to a temporary overloading or maintenance of the server, which is exactly what a failing readiness probe is signaling.

📋 Practical Task

Implementing a Multi-Dependency Readiness Probe for a User Profile Service

You are working on a User Profile Service that depends on two external systems: a PostgreSQL database (for user data) and a Redis cache (for session data). Currently, the service only has a basic "Hello World" health check.

Your Task:

• Modify the ReadinessHandler to check the health of both the Database and the Redis client.
• The handler must return 200 OK only if both dependencies are reachable.
• If either dependency fails, it must return 503 Service Unavailable.
• Ensure that both checks are wrapped in a context.WithTimeout of 1 second to prevent the health check from hanging.
• Implement a separate /livez endpoint that returns 200 OK regardless of dependency state.

Assume you have a db *sql.DB and a rdb *redis.Client available in your handler struct.