I've seen this play out in a dozen different production incidents: a developer pushes a new version of a service to a Kubernetes cluster using a standard RollingUpdate, and for a brief window of about 30 seconds, the error rate spikes. They check the logs, see a few "connection refused" errors, and conclude that the network is just "jittery" or the load balancer is slow to update. They assume the infrastructure is handling the transition, so the code shouldn't matter.

The Myth: "Kubernetes RollingUpdates are Automatically Zero-Downtime"

The misconception is that the orchestrator—whether it's K8s, Nomad, or AWS ECS—magically handles the handoff of traffic. It doesn't. If your Go service is just a standard http.ListenAndServe loop, the orchestrator sends a SIGTERM to your process, and your app immediately vanishes. Any request that was mid-flight—perhaps a critical payment transaction or a large file upload—is severed instantly. The client gets a 502 Bad Gateway or a connection reset. That is not zero-downtime; that's just "fast failure."

The Reality: Zero-Downtime is a Partnership Between Code and Infrastructure

To actually achieve zero-downtime, your Go code must be "signal-aware." You have to tell your application to stop accepting new requests but finish the ones it's already processing before finally exiting. I always tell my juniors: the orchestrator tells you when to leave, but your code decides how to leave.

The secret weapon here is http.Server.Shutdown(). Instead of letting the process be killed abruptly, you catch the termination signal and trigger a graceful shutdown. Here is a pattern I use for almost every production service I write:

package main

import (
	"context"
	"log"
	"net/http"
	"os"
	"os/signal"
	"syscall"
	"time"
)

func main() {
	server := &http.Server{Addr: ":8080"}

	// Run server in a goroutine so it doesn't block the signal listener
	go func() {
		if err := server.ListenAndServe(); err != http.ErrServerClosed {
			log.Fatalf("HTTP server ListenAndServe: %v", err)
		}
	}()

	// Create a channel to listen for OS signals
	stop := make(chan os.Signal, 1)
	signal.Notify(stop, os.Interrupt, syscall.SIGTERM)

	<-stop // Block until we receive a signal

	log.Println("Shutting down gracefully...")

	// Give the server 30 seconds to finish existing requests
	ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
	defer cancel()

	if err := server.Shutdown(ctx); err != nil {
		log.Fatalf("Server forced to shutdown: %v", err)
	}

	log.Println("Server stopped.")
}

Handling the "Readiness Gap" with PreStop Hooks

Even with the code above, you might still see a few errors. Why? Because there is a race condition between the time the orchestrator tells your pod to terminate and the time the load balancer removes your pod's IP from its rotation. If the load balancer sends a request 100ms after your app receives SIGTERM and stops accepting new connections, that request fails.

To fix this, you don't actually change the Go code—you change the deployment configuration. I recommend adding a preStop hook in your Kubernetes manifest that simply sleeps for 5 to 10 seconds. This forces the pod to wait while the load balancer propagates the "removal" of the pod before the Go app even begins its graceful shutdown sequence.

When Graceful Shutdown Isn't Enough: The Blue-Green Approach

If you're dealing with stateful connections (like WebSockets) or migrations that cannot coexist (like a breaking database schema change), a RollingUpdate—even a graceful one—won't cut it. In those cases, you move to a Blue-Green deployment.

• Blue: The current stable version.
• Green: The new version, deployed in parallel.

You route a small percentage of traffic to Green. If the metrics look good, you flip the switch and send 100% to Green. Only then do you decommission Blue. This adds cost because you're running double the infrastructure for a while, but it's the only way to ensure a safe rollback if the new version has a latent bug that only appears under load.

📋 Practical Task

Implementing a Signal-Aware Graceful Shutdown for a Transaction API

You have been handed a legacy "Payment Gateway" service that is currently causing 502 errors every time it is deployed. The service processes payments that can take anywhere from 2 to 10 seconds to complete.

Your Task: Modify the provided Go code to implement a graceful shutdown. The application must:

1. Catch SIGINT and SIGTERM signals.
2. Stop accepting new HTTP requests immediately upon receiving the signal.
3. Allow currently active payment requests to finish, but force a shutdown if they take longer than 15 seconds.
4. Log "Shutting down..." when the signal is received and "Clean exit" when the process finally terminates.

// STARTING CODE
package main

import (
	"fmt"
	"net/http"
	"time"
)

func paymentHandler(w http.ResponseWriter, r *http.Request) {
	// Simulate a slow payment process
	time.Sleep(5 * time.Second)
	fmt.Fprintln(w, "Payment Processed Successfully")
}

func main() {
	http.HandleFunc("/pay", paymentHandler)
	fmt.Println("Payment Gateway running on :8080")
	http.ListenAndServe(":8080", nil) 
}
// END STARTING CODE