I've been thinking about how we handle background tasks. Let's say we're building a notification system that sends out 1,000 emails when a big event happens. My first instinct—and probably yours too—is to just throw them all into goroutines and let Go's scheduler handle it. It seems like the "Go way," right?

The "Just Fire and Forget" Approach

Let's try that. I'll write a quick function to simulate sending an email and just loop through a slice of addresses, spawning a goroutine for every single one.

package main

import (
	"fmt"
	"time"
)

func sendEmail(id int, email string) {
	fmt.Printf("Sending email %d to %s...\n", id, email)
	time.Sleep(time.Millisecond * 500) // Simulate network lag
	fmt.Printf("Email %d sent!\n", id)
}

func main() {
	emails := []string{"user1@test.com", "user2@test.com", "user3@test.com", "user4@test.com"}
	for i, email := range emails {
		go sendEmail(i, email)
	}
	fmt.Println("All emails queued!")
}

If you run this, you'll notice something immediately: it prints "All emails queued!" and then the program exits. None of the emails actually get sent. I forgot that the main goroutine doesn't wait for the others. I could throw a time.Sleep at the end of main, but that's a hack and it's fragile. I don't know how long the emails will actually take.

But there's a bigger problem here. If I had 100,000 emails instead of four, I'd be spawning 100,000 goroutines simultaneously. While goroutines are cheap, they aren't free. More importantly, the external email API would probably rate-limit me or crash my connection for trying to open 100,000 sockets at once. I need a way to limit the concurrency.

Taming the Chaos with a Worker Pool

Instead of one goroutine per job, I want a fixed number of "workers" that stay alive and just pull jobs from a queue. In Go, that queue is naturally a channel. Let's pivot: I'll create a jobs channel and start, say, three worker goroutines that listen to it.

package main

import (
	"fmt"
	"time"
)

func worker(id int, jobs <-chan int) {
	for j := range jobs {
		fmt.Printf("Worker %d processing job %d\n", id, j)
		time.Sleep(time.Millisecond * 500)
	}
}

func main() {
	jobs := make(chan int, 100)

	// Start 3 workers
	for w := 1; w <= 3; w++ {
		go worker(w, jobs)
	}

	// Send 10 jobs
	for j := 1; j <= 10; j++ {
		jobs <- j
	}
	close(jobs)
	fmt.Println("Jobs dispatched!")
}

Okay, this is better. I've limited the concurrency to exactly three workers. But I'm back to the same problem as before: the program exits before the workers finish the jobs. I can see the "Jobs dispatched!" message, but the workers are cut off mid-sentence. I need a way to signal to main that the work is actually done.

Making Sure We Actually Finish

This is where sync.WaitGroup comes in. I need to tell the WaitGroup how many workers I'm waiting for, and each worker needs to signal when they've exited. But wait—the workers only exit when the jobs channel is closed. So the flow should be: dispatch jobs $\rightarrow$ close channel $\rightarrow$ wait for workers to finish.

Here is the refined version. I've added the WaitGroup and shifted the worker logic slightly.

package main

import (
	"fmt"
	"sync"
	"time"
)

func worker(id int, jobs <-chan int, wg *sync.WaitGroup) {
	defer wg.Done() // Signal completion when the function returns
	for j := range jobs {
		fmt.Printf("Worker %d started job %d\n", id, j)
		time.Sleep(time.Millisecond * 500)
		fmt.Printf("Worker %d finished job %d\n", id, j)
	}
}

func main() {
	const numJobs = 10
	const numWorkers = 3

	jobs := make(chan int, numJobs)
	var wg sync.WaitGroup

	// 1. Start the workers
	for w := 1; w <= numWorkers; w++ {
		wg.Add(1)
		go worker(w, jobs, &wg)
	}

	// 2. Send the jobs
	for j := 1; j <= numJobs; j++ {
		jobs <- j
	}
	
	// 3. Close the channel so workers know no more jobs are coming
	close(jobs)

	// 4. Wait for all workers to call wg.Done()
	wg.Wait()
	fmt.Println("All jobs completed successfully.")
}

Now it works exactly as intended. The workers process the jobs three at a time, and the main function patiently waits until the last worker finishes its final loop. I've effectively built a throttle for my system, protecting my downstream resources while still leveraging the power of concurrency.

📋 Practical Task

Build a Concurrent Image Thumbnail Processor

You are tasked with building a system that simulates processing images into thumbnails. Each "processing" job should take a random amount of time between 100ms and 1000ms to simulate varying image sizes.

Requirements:

• Create a Job struct that contains an ImageID (int) and a FileName (string).
• Implement a worker pool with exactly 5 workers.
• Create a list of 20 mock images to process.
• Use a sync.WaitGroup to ensure the program does not exit until all 20 images have been processed.
• Each worker should print which image it is starting and when it has finished.