Imagine you're running a busy coffee shop. You've got one person at the register taking orders. If that person also had to steam the milk, pull the espresso shots, and foam the latte, the line would stretch out the door. That's a bottleneck.

Instead, the register person (the source) writes the order on a cup and places it on a rail. Now, you have three baristas (the workers) who all grab cups from that same rail as fast as they can. This is Fan-Out: taking one stream of work and spreading it across multiple workers to get it done faster. But you can't have three different people handing drinks to customers at three different counters; that's chaos. So, all three baristas place the finished drinks on a single pick-up counter. This is Fan-In: gathering multiple streams of results back into one single channel for processing.

Spreading the Load with Fan-Out

In Go, Fan-Out is surprisingly simple. You create a channel to hold your "jobs" and then start multiple goroutines that all read from that same channel. Because Go channels are thread-safe, you don't have to worry about two workers grabbing the same job; the runtime handles that for you.

I've found that the trick here is deciding how many workers to spawn. If you're doing CPU-heavy work, matching the number of workers to runtime.NumCPU() is a good bet. If you're doing I/O (like hitting an API), you can usually push that number much higher.

type Result struct {
    url    string
    status int
}

func worker(urls <-chan string, results chan<- Result) {
    for url := range urls {
        // Simulate an HTTP request
        status := 200 
        results <- Result{url: url, status: status}
    }
}

func main() {
    urls := make(chan string, 10)
    results := make(chan Result, 10)

    // Fan-out: Start 3 workers
    for i := 0; i < 3; i++ {
        go worker(urls, results)
    }
    
    // Send work
    sites := []string{"google.com", "golang.org", "github.com", "stackoverflow.com"}
    for _, s := range sites {
        urls <- s
    }
    close(urls) // Tell workers there's no more work
}

Gathering the Results with Fan-In

The code above has a problem: the results channel is being fed by three different goroutines. If we just try to range over it, we might hit a deadlock or miss data because we don't know when all the workers are actually finished. We need a way to "merge" these streams.

The most robust way to handle Fan-In is using a sync.WaitGroup. I usually wrap the worker management in a separate goroutine so the main thread can just listen to the merged results without blocking.

func merge(cs ...<-chan Result) <-chan Result {
    var wg sync.WaitGroup
    out := make(chan Result)

    // Function to move values from a source channel to the output channel
    output := func(c <-chan Result) {
        for n := range c {
            out <- n
        }
        wg.Done()
    }

    wg.Add(len(cs))
    for _, c := range cs {
        go output(c)
    }

    // This is the secret sauce: close the out channel 
    // only after ALL source channels are drained
    go func() {
        wg.Wait()
        close(out)
    }()

    return out
}

Putting it Together: The Pipeline

When you combine these, you get a powerful pipeline. You produce work, fan it out to workers, and fan the results back in. The key is the ownership of the channels. The producer closes the jobs channel, the workers finish their loops, and the merge function closes the results channel once the WaitGroup hits zero.

It looks like a lot of "plumbing" at first, but this pattern is exactly how high-performance Go services handle massive streams of data. You're essentially building a conveyor belt where the middle section can be widened (more workers) depending on how slow the processing step is compared to the input step.

📋 Practical Task

Exercise: Building a Concurrent Website Status Scanner

Your task is to create a program that checks the availability of a list of websites concurrently. You'll need to implement both the Fan-Out and Fan-In patterns to ensure the program is efficient and shuts down cleanly.

Requirements:

• Create a slice of at least 10 URLs.
• Implement a jobs channel that distributes these URLs to 5 worker goroutines (Fan-Out).
• Each worker should simulate a network request (use time.Sleep for a random duration between 100ms and 500ms) and return a Result struct containing the URL and a fake status code.
• Implement a merge function (Fan-In) that collects results from all workers into a single output channel.
• The main function must range over the final results channel and print each result, exiting only after all websites have been processed.
• Ensure there are no goroutine leaks; all channels must be closed appropriately.