I’ve seen this a lot when developers first move from synchronous languages to Go: they treat goroutines like "free" magic. They think that if they have 1,000 URLs to fetch, the fastest way to do it is to launch 1,000 goroutines immediately. It feels intuitive. Why wait for one request to finish when you can start them all at once?

The "Spawn and Pray" Fallacy

Here is why that approach fails in the real world. If you write a loop that just calls go fetch(url) for a massive list, you aren't actually optimizing for speed; you're creating a bottleneck at the OS level. You'll likely hit the too many open files error because every network connection is a file descriptor. Even if your OS handles it, the target servers might see 1,000 simultaneous hits from a single IP and flag you as a DDoS attack, promptly dropping your packets.

// DON'T DO THIS with large lists
for _, url := range urls {
    go func(u string) {
        fetch(u) // This is a recipe for a crash or a ban
    }(url)
}

Managing Pressure with Worker Pools

The correct way to handle this is to decouple the work to be done from the workers doing it. Instead of one goroutine per URL, we create a fixed number of workers—say, 10 or 20—that live for the duration of the program. These workers listen to a "jobs" channel. As soon as a worker finishes one fetch, it grabs the next available URL from the channel.

I prefer this pattern because it gives you a "knob" to turn. If you find the network is saturated, you turn the worker count down. If you have a massive pipe and the servers can handle it, you turn it up. You're controlling the pressure on the system rather than just letting it explode.

To get this working, you'll need three main components:

• A Jobs Channel: A buffered channel where you send the URLs you want fetched.
• A Results Channel: A way for workers to send back the status (or error) of the fetch.
• A Sync Mechanism: A sync.WaitGroup to ensure the main program doesn't exit before the workers finish their final tasks.

One little tip: remember to close your jobs channel. If your workers are using a for url := range jobs loop, they will only stop once the channel is closed. If you forget this, your workers will hang forever, leaking memory and keeping your program from ever completing.

📋 Practical Task

Build a Limited-Concurrency Site Health Checker

Your task is to build a tool that checks the availability of a list of websites without overwhelming the network. Follow these specific requirements:

• Input: Create a slice of at least 10 different URLs (use real sites or placeholders like http://google.com).
• Concurrency Limit: Implement a worker pool with exactly 3 worker goroutines. No matter how many URLs are in your list, only 3 should ever be fetching at the same time.
• The Logic: Each worker should pull a URL from a jobs channel, perform an http.Get, and send a result (the URL and whether it returned a 200 OK status) into a results channel.
• Synchronization: Use a sync.WaitGroup to ensure all workers complete their tasks before the program prints the final summary and exits.
• Output: The main function should collect all results from the channel and print them in the format: [URL] - Status: Up/Down.

Bonus challenge: Add a timeout to the http.Client so that a single hanging server doesn't block one of your workers indefinitely.