When you first start working with multi-threading in Python, your instinct is usually to reach for the tools you already know. If you need a way to pass work from a main thread to a pool of workers, a list feels like the obvious choice. You just append a task to the end and pop it off the front. It's simple, it's intuitive, and for a while, it actually seems to work.

The temptation of the simple list

Imagine we're building a basic web crawler. We have a list of URLs to visit, and we want to spin up five threads to download them concurrently so we aren't waiting on one slow server to respond before moving to the next. In a naive implementation, it looks something like this:

import threading
import time

# A simple list to hold our URLs
url_queue = ["http://site1.com", "http://site2.com", "http://site3.com", "http://site4.com"]

def worker():
    while len(url_queue) > 0:
        # The naive pop
        url = url_queue.pop(0)
        print(f"Downloading {url}...")
        time.sleep(1) # Simulate network latency

threads = [threading.Thread(target=worker) for _ in range(5)]
for t in threads: t.start()
for t in threads: t.join()

If you run this with a small list, it might work perfectly. You might even convince yourself that since Python has a Global Interpreter Lock (GIL), this is safe. But that's a dangerous assumption. The GIL protects the internal state of the Python interpreter, but it doesn't protect your application's logic.

Where the race condition hides

The problem here is a classic "check-then-act" race condition. Look closely at the while len(url_queue) > 0 line followed by url_queue.pop(0). These are two separate operations.

Imagine there is only one URL left in the list. Thread A checks the length, sees it's 1, and enters the loop. Before Thread A can actually call pop(0), the operating system pauses it and lets Thread B run. Thread B also checks the length, sees it's 1, and enters the loop. Now, Thread A wakes up and pops the last item. Then Thread B wakes up and tries to pop the last item—but the list is now empty. IndexError: pop from empty list. Boom. Your crawler crashes in production, and because it only happens when the timing is just right, you'll spend three hours trying to reproduce it locally.

Letting queue.Queue handle the heavy lifting

To do this right, we need a data structure that makes the "check and pop" action atomic. This is exactly why the queue module exists. A queue.Queue isn't just a list with a different name; it's specifically engineered for multi-threaded producers and consumers. It handles all the underlying locking for you, so you don't have to manually wrap everything in threading.Lock().

import threading
import queue
import time

# Use a thread-safe Queue instead of a list
url_queue = queue.Queue()
for url in ["http://site1.com", "http://site2.com", "http://site3.com", "http://site4.com"]:
    url_queue.put(url)

def worker():
    while True:
        try:
            # block=False tells the queue to raise queue.Empty if nothing is there
            url = url_queue.get(block=False)
            print(f"Downloading {url}...")
            time.sleep(1)
            url_queue.task_done() # Signal that the task is complete
        except queue.Empty:
            break

threads = [threading.Thread(target=worker) for _ in range(5)]
for t in threads: t.start()
for t in threads: t.join()

The beauty of url_queue.get(block=False) is that it combines the check and the retrieval into one thread-safe operation. Either you get an item, or you get an exception; there is no "in-between" state where another thread can sneak in and steal the item from under you.

I should also mention task_done() and queue.join(). In a real-world system, you often don't know when the queue will be empty because workers might be adding new URLs to the queue as they find them on pages. Instead of checking the length, you can call url_queue.join() in your main thread. This blocks the main thread until every item put into the queue has been processed and marked with task_done(). It's a much cleaner way to coordinate the shutdown of your worker pool than trying to track the list size manually.

📋 Practical Task

Exercise: Building a Concurrent Image Metadata Extractor

You need to create a program that simulates extracting metadata from a large batch of image files. Since simulating I/O (like reading files) is the goal, you will use time.sleep() to represent the processing time.

Requirements:

• Create a queue.Queue and populate it with a list of 20 dummy filenames (e.g., "image_1.jpg", "image_2.jpg", ...).
• Implement a worker function that pulls a filename from the queue, simulates a 0.1 to 0.5 second delay (use random.uniform), and prints "Processed [filename]".
• Ensure the worker calls task_done() after processing each image.
• Start 4 worker threads.
• Instead of using a while True loop with try/except queue.Empty, use the queue.join() method in your main thread to wait for all images to be processed.
• Once join() returns, print "All images processed. Shutting down." and terminate your worker threads (Hint: you can use daemon threads for the workers so they exit automatically when the main thread ends).