I've always found that the best way to understand how a message queue actually works is to try and build a "naive" version first and watch it fail. We're not going to use RabbitMQ or Kafka here; we're going to build a simple, in-memory queue to handle a common problem: sending emails. Sending an email is slow—it involves network calls and external APIs—and we can't let that slow down our main application loop.

The basic channel approach

My first instinct is usually to just throw a channel at the problem. It's the "Go way," right? Let's see what happens when I set up a simple producer and a consumer.

package main

import (
	"fmt"
	"time"
)

func main() {
	emails := make(chan string, 10)

	// The consumer
	go func() {
		for msg := range emails {
			fmt.Printf("Sending email to %s...\n", msg)
			time.Sleep(time.Second) // Simulate network lag
			fmt.Printf("Done sending to %s\n", msg)
		}
	}()

	// The producer
	emails <- "user1@example.com"
	emails <- "user2@example.com"
	fmt.Println("Emails queued!")
}

If you run this, you'll probably see "Emails queued!" and then the program immediately exits. You might see one email start to send, but the program dies before the worker finishes. This is a classic mistake. The main goroutine doesn't care that the worker goroutine is still churning away; once main returns, everything stops. We need a way to tell the program, "Wait until the work is actually done."

Managing the lifecycle with WaitGroups

To fix the abrupt exit, I'll bring in sync.WaitGroup. But I also realize that just using a raw channel in main is messy. If I want to use this queue in multiple parts of my app, I should wrap it in a struct. Let's try to formalize this into a Queue type.

type EmailJob struct {
	Email   string
	Content string
}

type MessageQueue struct {
	jobs chan EmailJob
	wg   sync.WaitGroup
}

func NewQueue(bufferSize int) *MessageQueue {
	return &MessageQueue{
		jobs: make(chan EmailJob, bufferSize),
	}
}

func (q *MessageQueue) StartWorker() {
	q.wg.Add(1)
	go func() {
		defer q.wg.Done()
		for job := range q.jobs {
			fmt.Printf("Processing email to %s\n", job.Email)
			time.Sleep(time.Millisecond * 500)
		}
	}()
}

func (q *MessageQueue) Enqueue(job EmailJob) {
	q.jobs <- job
}

func (q *MessageQueue) Stop() {
	close(q.jobs)
	q.wg.Wait()
}

Now we're getting somewhere. By adding Stop(), I can close the channel, which signals the range loop in the worker to finish processing whatever is left in the buffer and then exit. The wg.Wait() ensures the main thread hangs on until the worker is truly finished.

Handling the "Full Buffer" panic

Here's where things get interesting. I set the buffer size to 10. What happens if my producer is faster than my consumer and I try to send the 11th email? The program will block. In a real web server, that means a user's HTTP request just hangs indefinitely until the queue clears. That's a terrible user experience.

I want the queue to be "non-blocking" or at least give me a way to handle a full queue. I'll use a select block with a default case. This allows me to attempt a send, and if the channel is full, I can return an error immediately instead of freezing the thread.

func (q *MessageQueue) Enqueue(job EmailJob) error {
	select {
	case q.jobs <- job:
		return nil
	default:
		return fmt.Errorf("queue is full, could not enqueue job for %s", job.Email)
	}
}


I've found that this pattern is essential for building resilient systems. You have to decide: do I want to block the producer (backpressure), or do I want to drop the message and report a failure? For emails, reporting a "system busy" error is usually better than crashing the entire request pipeline.

Adding multiple workers for throughput
One worker is slow. But since we're using a channel, we can just spin up multiple workers reading from that same channel. Go's channel implementation handles the orchestration for us—only one worker will receive any given job.

I'll adjust the StartWorker logic to take a count. I'm noticing that if I start 5 workers, the emails get processed roughly 5x faster. The only thing to watch out for is the order of execution; because goroutines are scheduled non-deterministically, "Email 1" might finish after "Email 2". For a message queue, that's usually an acceptable trade-off for the massive increase in throughput.

📋 Practical Task

Exercise: Building a Durable Email Dispatcher with Retries

In the lesson, we built a queue that either succeeds or fails immediately if the buffer is full. In a real-world scenario, some jobs fail not because the queue is full, but because the external API (the email provider) is temporarily down.

Your task is to extend the MessageQueue implementation with the following requirements:

• Retry Logic: Modify the EmailJob struct to include a RetryCount integer.
• Failure Simulation: In the worker loop, simulate a random failure (e.g., using rand.Intn(3) == 0). If a job "fails," the worker should not just discard it.
• Re-queuing: If a job fails and its RetryCount is less than 3, increment the count and send it back into the queue.
• Dead Letter Handling: If a job fails and has reached the maximum of 3 retries, print a specific error message: "Job failed permanently for [email]".

Ensure that your Stop() method still allows the queue to drain completely, including any jobs that were re-queued for retries.