How do I avoid spawning a thread for every single file?

If you're processing a directory with ten thousand small log files, calling std::thread::spawn ten thousand times is a great way to make your OS very unhappy. Thread creation is expensive, and context switching will eventually eat all your performance gains.

In a real-world production system, I'd tell you to just use Rayon, but since we're building this to understand the plumbing, the best approach is a worker pool pattern. You spawn a fixed number of threads (usually matching your CPU core count) and have them pull filenames from a shared queue. I personally prefer using a channel for this—it's the "Rust way" to communicate by sharing memory.

use std::sync::mpsc;
use std::thread;

fn main() {
    let files = vec!["log1.txt", "log2.txt", "log3.txt", "log4.txt"]; 
    let (tx, rx) = mpsc::channel();
    
    // Give the receiver to the workers
    let rx = std::sync::Arc::new(std::sync::Mutex::new(rx));

    let mut handles = vec![];
    for id in 0..4 { // Fixed pool of 4 workers
        let rx = std::sync::Arc::clone(&rx);
        let handle = thread::spawn(move || {
            loop {
                // We lock the receiver to grab one filename at a time
                let file = {
                    let lock = rx.lock().unwrap();
                    lock.recv().ok() 
                };

                match file {
                    Some(f) => println!("Worker {} processing {}", id, f),
                    None => break, // Channel closed, worker exits
                }
            }
        });
        handles.push(handle);
    }

    for f in files {
        tx.send(f).unwrap();
    }
    drop(tx); // Close the channel so workers know to stop

    for h in handles { h.join().unwrap(); }
}

How do I collect results back into a single total without a bottleneck?

You'll be tempted to wrap your global counter in an Arc<Mutex<usize>> and have every thread lock it every time they find a match. Don't do that. If your threads are finding matches frequently, they'll spend more time fighting over the lock (contention) than actually processing files.

The trick is to let each thread maintain its own local count and only send the final result back to the main thread once the file is finished. I usually set up a second channel specifically for the results. This keeps the "hot path" lock-free.

// Inside the worker thread loop...
let mut local_count = 0;
let content = std::fs::read_to_string(file_path).unwrap();
for line in content.lines() {
    if line.contains("ERROR") {
        local_count += 1;
    }
}
// Send the aggregate for this file back to the collector
result_tx.send(local_count).unwrap();

What happens if one file is corrupted? Do I lose everything?

This is where a lot of people trip up. If you use .unwrap() inside a thread and it panics, that thread dies. If you aren't careful, you might not even notice it happened until you realize your final count is lower than expected.

The JoinHandle returned by thread::spawn actually returns a Result. If the thread panicked, the Err variant contains the panic payload. However, a cleaner pattern is to make your thread closure return a Result<usize, std::io::Error>. This way, the main thread can decide whether a single missing file should crash the whole program or just be logged as a warning.

I always recommend mapping your errors to a custom type or using anyhow if the project is large, but for a processor like this, simply returning the io::Result from the thread and handling it during the join() phase is the most robust approach.

📋 Practical Task

Build a Parallel Log-Level Aggregator

Create a program that processes a list of simulated log files (you can create these as strings or temporary files). The program must:

• Spawn a worker pool based on the number of logical CPUs on the machine (use std::thread::available_parallelism).
• Distribute the files across these workers using a channel.
• Each worker must count three different levels: INFO, WARN, and ERROR.
• Instead of a single integer, the workers should send a HashMap<String, usize> (or a custom struct) back to the main thread containing the counts for that specific file.
• The main thread must aggregate these maps into a final global total and print the summary.
• Ensure that if a file cannot be read, the worker logs the error to stderr but continues processing the remaining files in the queue.