When you first start dealing with large API responses in Go, the temptation is to treat data like a physical object: you grab it all, put it in a box, wrap that box in compression, and then ship it. It feels intuitive, but in a production environment, this "buffer-first" approach is a recipe for memory exhaustion.

The memory trap of buffer-first compression

I've seen this pattern countless times in code reviews. A developer wants to compress a large JSON response, so they marshal the data into a byte slice, create a bytes.Buffer, and wrap that buffer in a gzip.Writer. It looks something like this:

func naiveHandler(w http.ResponseWriter, r *http.Request) {
    data := fetchLargeProductCatalog() // Imagine this is 10MB of JSON
    
    var buf bytes.Buffer
    gz := gzip.NewWriter(&buf)
    
    // We write the data to the compressor, which writes to the buffer
    gz.Write(data)
    gz.Close() 
    
    w.Header().Set("Content-Encoding", "gzip")
    w.Write(buf.Bytes())
}

On the surface, it works. But look at what's happening in your RAM. You have the raw product catalog in memory, you have the bytes.Buffer growing as it stores the compressed version, and you're allocating a brand new gzip.Writer for every single request. If you have 100 concurrent users requesting a 10MB catalog, you aren't just using 1GB of RAM—you're putting immense pressure on the Garbage Collector because of all those short-lived, large allocations. It's inefficient and, frankly, dangerous.

Streaming through the writer

The "Go way" is to embrace the io.Writer interface. Instead of creating an intermediate buffer, you should wrap the http.ResponseWriter itself. This creates a pipeline where data flows from your data source, through the compressor, and directly onto the network socket. The data never needs to exist as one giant compressed blob in your memory.

Here is how I would actually implement this. Notice how we chain the json.Encoder directly into the gzip.Writer:

func streamingHandler(w http.ResponseWriter, r *http.Request) {
    products := fetchProductCatalog() 
    
    w.Header().Set("Content-Type", "application/json")
    w.Header().Set("Content-Encoding", "gzip")
    
    // Wrap the response writer directly
    gz := gzip.NewWriter(w)
    
    // Very important: gzip.Writer must be closed to flush the trailer
    defer gz.Close()
    
    // Stream the JSON encoding directly into the gzip writer
    if err := json.NewEncoder(gz).Encode(products); err != nil {
        // Handle error (though headers are already sent)
        return
    }
}

The difference here is massive. By using json.NewEncoder(gz), the JSON library writes small chunks of data to the gzip writer, which in turn writes small chunks to the network. Your memory footprint stays flat regardless of whether the catalog is 1MB or 100MB.

The hidden cost of the compressor

There is one more detail that often trips people up: gzip.NewWriter is surprisingly expensive. It allocates a significant amount of internal state to manage the compression dictionary. If your service is handling thousands of requests per second, creating a new writer every time will spike your CPU and memory usage.

In a high-performance system, you wouldn't use NewWriter inside the handler. Instead, you'd use a sync.Pool to reuse gzip.Writer instances. You'd pull a writer from the pool, use gz.Reset(w) to point it at the current response writer, and then put it back in the pool when the request is finished. I won't clutter this lesson with the pool implementation, but keep it in the back of your mind: whenever you see NewWriter in a hot loop or a high-traffic handler, ask yourself if you can Reset an existing one instead.

📋 Practical Task

Exercise: Building a Gzip-Aware Product Catalog API

Your task is to create a Go HTTP server that serves a large list of simulated "Product" structs. You must ensure that the server does not load the entire compressed payload into memory before sending it.

• Define a Product struct with fields: ID (int), Name (string), and Price (float64).
• Create a handler that generates a slice of 1,000 products.
• Implement the handler using the streaming approach: wrap the http.ResponseWriter in a gzip.Writer and use a json.Encoder to write the data.
• Ensure the Content-Encoding: gzip and Content-Type: application/json headers are set correctly.
• Verify that you have properly closed the gzip.Writer to ensure the compression trailer is sent.