A few years back, I was on call for a service that handled payment webhooks. Everything looked "green" on the dashboard—the CPU was low, the memory was stable, and the health check endpoint was returning 200 OK. But our customer support Slack channel was exploding because users weren't seeing their credits updated. When I finally dug into the logs, I found that 15% of requests were timing out after 30 seconds, but because they were timing out at the load balancer level, the application logs were practically silent. I was flying blind. I didn't need more logs; I needed a way to see the distribution of request latencies in real-time. That's exactly why we instrument with Prometheus.

Choosing Your Metric Type

In the Go ecosystem, the prometheus/client_golang library is the gold standard. Before you start adding code, you have to decide which "tool" fits the data you're tracking. If you pick the wrong one, your queries in Grafana will be a nightmare.

First, you have Counters. These only ever go up. Think of them like an odometer in a car. Use these for things like http_requests_total or orders_processed_total. You don't care what the absolute number is as much as you care about the rate of increase. If the rate of 500-errors spikes, you have a problem.

Then there are Gauges. These can go up and down. Use these for "current state" measurements: the number of active Goroutines, the current size of a worker queue, or memory usage. If you're tracking how many users are currently online, that's a Gauge.

Finally, there are Histograms. These are the most powerful and the most misunderstood. A histogram doesn't store every single value; instead, it counts how many observations fall into predefined "buckets." This is how we track latencies. You don't want the average latency—averages hide the pain of your slowest users. You want the 99th percentile (P99), and histograms make that possible.

Wiring Prometheus into your Go Service

To actually get these metrics out of your app and into a Prometheus server, you need to expose a scrape endpoint—usually /metrics. Prometheus is a "pull" system; it hits your app every few seconds and asks, "What are your current numbers?"

package main

import (
	"net/http"
	"time"

	"github.com/prometheus/client_golang/prometheus"
	"github.com/prometheus/client_golang/prometheus/promauto"
	"github.com/prometheus/client_golang/prometheus/promhttp"
)

// I use promauto to handle the registration automatically. 
// It saves you from having to call prometheus.MustRegister() manually.
var (
	opsProcessed = promauto.NewCounter(prometheus.CounterOpts{
		Name: "myapp_processed_ops_total",
		Help: "The total number of processed operations",
	})

	requestDuration = promauto.NewHistogramVec(prometheus.HistogramOpts{
		Name:    "myapp_request_duration_seconds",
		Help:    "Latency of requests in seconds",
		Buckets: []float64{.005, .01, .025, .05, .1, .25, .5, 1, 2.5, 5, 10},
	}, []string{"handler"})
)

func handler(w http.ResponseWriter, r *http.Request) {
	start := time.Now()
	
	// Simulate some work
	time.Sleep(100 * time.Millisecond)
	
	opsProcessed.Inc()
	requestDuration.WithLabelValues("main_handler").Observe(time.Since(start).Seconds())
	w.Write([]byte("Done!"))
}

func main() {
	http.HandleFunc("/do-work", handler)
	// This is the magic endpoint Prometheus scrapes
	http.Handle("/metrics", promhttp.Handler())
	http.ListenAndServe(":8080", nil)
}

The Power of Labels and the Danger of Cardinality

You'll notice I used NewHistogramVec instead of just NewHistogram. The "Vec" stands for Vector, which allows you to add labels. Labels are essentially dimensions. Instead of creating request_duration_login and request_duration_logout, you create one metric with a label handler="login".

But a word of caution: be extremely careful with label values. This is where most engineers shoot themselves in the foot. If you put a User ID or a Request ID as a label value, you create "high cardinality." Prometheus creates a new time series for every unique combination of labels. If you have 100,000 users and you put user_id in a label, you'll crash your Prometheus server by blowing out its memory. Keep label values bounded—think HTTP status codes, region names, or service versions.

📋 Practical Task

Instrumenting a Payment Gateway Simulation

You are tasked with adding observability to a simulated payment gateway. The service currently processes payments but provides no insight into its performance or error rates.

Requirements:

• Create a Counter named payment_attempts_total that tracks every payment attempt. It must have a label called status (values: "success" or "failure").
• Create a Gauge named active_payment_connections that increments when a payment starts and decrements when it finishes.
• Create a Histogram named payment_processing_seconds that tracks the time it takes to process a payment, using buckets: 0.1, 0.5, 1.0, 2.0, 5.0.
• Expose these via the /metrics endpoint on port :8080.

Starter Logic: Your handler should randomly simulate a payment taking between 50ms and 3s, and randomly fail about 20% of the time to ensure your status="failure" counter is actually being tested.