Imagine you're running a busy coffee shop. Every time a customer walks in, you don't go out to the street, hire a brand-new barista on the spot, teach them where the espresso machine is, and then fire them the second the latte is handed over. That would be an absolute nightmare. The time spent hiring and onboarding would take longer than actually making the coffee.

Instead, you hire a fixed team of, say, four baristas. They stay behind the counter all day. When a customer arrives, their order goes onto a ticket rail (a queue). The first available barista grabs the next ticket, makes the drink, and then looks back at the rail for the next task. If there are no orders, they just hang out and wait.

In C++, this is exactly what a Thread Pool does. Spinning up a std::thread isn't free; it requires a syscall to the OS to allocate a stack and manage kernel resources. If you're doing thousands of tiny tasks—like processing pixels in an image or handling incoming network packets—creating a thread for every single one will actually slow your program down due to "thread thrashing."

The High Cost of Constant Hiring

I've seen a lot of developers try to solve concurrency by just throwing threads at the problem in a for loop. While that works for two or three heavy tasks, it falls apart at scale. A thread pool solves this by decoupling the task (the work to be done) from the worker (the thread doing it).

To build this, we need three main components:

• A vector of worker threads that live for the duration of the pool's life.
• A queue of tasks (usually std::function<void()>) waiting to be executed.
• A synchronization mechanism (a mutex and a condition variable) so workers know when a new task has arrived without spinning their CPUs at 100% while idle.

Managing the Order Queue

The trickiest part is making sure the workers don't fight over the queue. We use a std::mutex to protect the queue and a std::condition_variable to put threads to sleep when the queue is empty. I prefer using std::future and std::packaged_task here, because in the real world, you rarely want a task to just "run"—you usually want a result back from it.

#include <vector>
#include <queue>
#include <thread>
#include <mutex>
#include <condition_variable>
#include <functional>
#include <future>

class ThreadPool {
    std::vector<std::thread> workers;
    std::queue<std::function<void()>> tasks;
    
    std::mutex queue_mutex;
    std::condition_variable condition;
    bool stop = false;

public:
    ThreadPool(size_t threads) {
        for(size_t i = 0; i < threads; ++i) {
            workers.emplace_back([this] {
                for(;;) {
                    std::function<void()> task;
                    {
                        std::unique_lock<std::mutex> lock(this->queue_mutex);
                        this->condition.wait(lock, [this]{ 
                            return this->stop || !this->tasks.empty(); 
                        });
                        if(this->stop && this->tasks.empty()) return;
                        task = std::move(this->tasks.front());
                        this->tasks.pop();
                    }
                    task();
                }
            });
        }
    }

    template<class F, class... Args>
    auto enqueue(F&& f, Args&&... args) 
     -> std::future<typename std::invoke_result<F, Args...>::type> 
    {
        using return_type = typename std::invoke_result<F, Args...>::type;

        auto task = std::make_shared<std::packaged_task<return_type()>>(
            std::bind(std::forward<F>(f), std::forward<Args>(args)...)
        );
        
        std::future<return_type> res = task->get_future();
        {
            std::unique_lock<std::mutex> lock(queue_mutex);
            tasks.emplace([task](){ (*task)(); });
        }
        condition.notify_one();
        return res;
    }

    ~ThreadPool() {
        {
            std::unique_lock<std::mutex> lock(queue_mutex);
            stop = true;
        }
        condition.notify_all();
        for(std::thread &worker: workers) worker.join();
    }
};

Gracefully Closing Shop

Notice the destructor. You can't just let the pool vanish; you have to tell the workers that the shop is closing (stop = true) and wake them all up (notify_all()). If you don't join the threads, your program will crash with a std::terminate because the threads are still running when the pool object is destroyed. It's the C++ equivalent of locking the doors and making sure everyone has clocked out before you turn off the lights.

One last tip: don't just set your pool size to 100 because you think "more is better." Usually, the sweet spot is the number of hardware cores available on the machine (std::thread::hardware_concurrency()). Adding more threads than cores just forces the OS to context-switch, which brings us right back to the performance overhead we were trying to avoid.

📋 Practical Task

Building a Parallel Heavy-Compute Simulator

To put this into practice, you're going to build a simulation of a batch-processing system. Your goal is to process a large set of "data packets" where each packet requires a simulated heavy calculation (use std::this_thread::sleep_for to mimic a 10-50ms compute task).

Requirements:

• Implement the ThreadPool class as discussed in the lesson.
• Create a vector of 100 "work items" (e.g., integers from 1 to 100).
• Enqueue these items into the pool. Each task should take the integer, "process" it (sleep), and return the square of that integer.
• Store the std::future objects returned by enqueue in a vector.
• Iterate through your futures to collect and print the results, ensuring that the main thread waits for all tasks to complete.
• Compare the total execution time using a single-threaded loop versus your ThreadPool (using std::chrono).