You've likely used std::thread for simple background work, but in a production environment, spawning a new thread for every single task is a recipe for disaster. The overhead of creating and destroying threads is expensive, and if you launch a thousand threads at once, your OS will spend more time context-switching than actually executing your code. That's why we use a Thread Pool: we create a fixed number of workers once, and then feed them tasks from a queue.

Setting up the Backbone

First, we need a way to store tasks. Since a task can be any callable object (a lambda, a function pointer, or a functor), std::function is our best bet. I'll wrap this in a class that manages a vector of workers and a queue of these tasks.

#include <vector>
#include <queue>
#include <thread>
#include <mutex>
#include <condition_variable>
#include <functional>
#include <future>

class ThreadPool {
    std::vector<std::thread> workers;
    std::queue<std::function<void()>> tasks;
    
    std::mutex queue_mutex;
    std::condition_variable condition;
    bool stop = false;

public:
    ThreadPool(size_t threads);
    ~ThreadPool();
    void enqueue(std::function<void()> task);
};

Spawning the Workers

In the constructor, we don't just create threads; we give them a loop. Each worker should wait until there is a task in the queue, grab it, execute it, and then go back to waiting. I prefer using a std::unique_lock here because the condition_variable needs it to atomically release the mutex while the thread sleeps.

ThreadPool::ThreadPool(size_t threads) {
    for(size_t i = 0; i < threads; ++i) {
        workers.emplace_back([this] {
            for(;;) {
                std::function<void()> task;
                {
                    std::unique_lock<std::mutex> lock(this->queue_mutex);
                    this->condition.wait(lock, [this]{ 
                        return this->stop || !this->tasks.empty(); 
                    });
                    
                    if(this->stop && this->tasks.empty()) return;
                    
                    task = std::move(this->tasks.front());
                    this->tasks.pop();
                }
                task();
            }
        });
    }
}

Fixing the "Zombie Thread" Problem

When I first wrote this pattern years ago, I made a classic mistake. I wrote the destructor like this:

// WRONG VERSION
~ThreadPool() {
    {
        std::lock_guard<std::mutex> lock(queue_mutex);
        stop = true;
    }
    for(std::thread & worker: workers) worker.join();
}

I quickly realized my program was hanging on exit. Why? Because the worker threads were stuck in condition.wait(). Setting stop = true is great, but the threads are asleep; they aren't checking the stop flag. They'll never wake up to see that they should exit, and join() will wait forever. I forgot to notify them.

Here is the corrected destructor. I added notify_all() to wake up every single worker so they can check the stop flag and exit gracefully:

~ThreadPool() {
    {
        std::lock_guard<std::mutex> lock(queue_mutex);
        stop = true;
    }
    condition.notify_all(); // Wake up all threads to let them finish
    for(std::thread & worker: workers) worker.join();
}

Returning Values with Futures

A void() task is fine for "fire and forget" work, but usually, you want a result back. To do this, we can't just use enqueue(std::function<void()> task). Instead, we can use a template and std::packaged_task. This allows us to return a std::future to the caller.

template<class F, class... Args>
auto enqueue(F&& f, Args&&... args) 
    → std::future<typename std::invoke_result<F, Args...>::type> 
{
    using return_type = typename std::invoke_result<F, Args...>::type;

    auto task = std::make_shared<std::packaged_task<return_type()>>(
        std::bind(std::forward<F>(f), std::forward<Args>(args)...)
    );
    
    std::future<return_type> res = task->get_future();
    {
        std::lock_guard<std::mutex> lock(queue_mutex);
        tasks.emplace([task](){ (*task)(); });
    }
    condition.notify_one();
    return res;
}

Now we can actually use this to do something concrete, like calculating the squares of a bunch of numbers in parallel:

int main() {
    ThreadPool pool(4);
    std::vector<std::future<int>> results;

    for(int i = 0; i < 8; ++i) {
        results.emplace_back(pool.enqueue([i] {
            return i * i;
        }));
    }

    for(auto && res : results) {
        printf("Result: %d\n", res.get());
    }
}

📋 Practical Task

Exercise: Implementing a Graceful Drain Shutdown

Currently, the ThreadPool destructor sets stop = true and immediately notifies all threads. If there are still tasks left in the queue when the destructor is called, those tasks might be discarded because the worker loop checks if(this->stop && this->tasks.empty()) return;, but the destructor doesn't guarantee that the queue is drained before the pool is destroyed.

Your Task: Modify the ThreadPool destructor or add a new method called wait_until_empty() that blocks the main thread until all currently enqueued tasks have been completed. Ensure that the pool does not shut down until the tasks queue is completely empty, regardless of whether the stop flag has been set.