I've been working on a script to process a huge batch of high-resolution images, and since image processing is CPU-intensive, I decided to use the multiprocessing module to take advantage of all my CPU cores. I wanted a simple way to keep track of how many images were finished in real-time, so I started with the most obvious approach: a global counter.

The Global Variable Trap

Here is exactly how I started. I defined a global variable completed_count, spawned a few processes to do some "work," and then printed the total. It looked logically sound to me at the time.

import multiprocessing
import time

completed_count = 0

def process_image(image_id):
    global completed_count
    print(f"Processing image {image_id}...")
    time.sleep(0.1) # Simulating work
    completed_count += 1

if __name__ == "__main__":
    processes = []
    for i in range(5):
        p = multiprocessing.Process(target=process_image, args=(i,))
        processes.append(p)
        p.start()

    for p in processes:
        p.join()

    print(f"Total images processed: {completed_count}")

When I ran this, I expected to see "Total images processed: 5". Instead, it printed "Total images processed: 0". Every single time. I stared at the code for a minute, thinking I'd missed a return statement or a scope issue, but the reality is more fundamental: processes do not share memory.

When you start a new process in Python, it gets its own entirely separate instance of the Python interpreter and its own memory space. The completed_count variable was copied into each child process. Each process incremented its own local copy of the counter, and when the process died, that local memory vanished. The parent process's counter never budged.

Giving Processes a Common Ground

To fix this, I need a way to tell Python, "Don't give these processes their own copy; give them a pointer to a piece of memory they can all see." For simple types, the multiprocessing module provides Value and Array. These are stored in shared memory outside the usual Python object heap.

Let's try swapping that global integer for a multiprocessing.Value.

import multiprocessing
import time

def process_image(image_id, counter):
    print(f"Processing image {image_id}...")
    time.sleep(0.1)
    # We have to access the actual value through the .value attribute
    counter.value += 1

if __name__ == "__main__":
    # 'i' stands for a signed integer
    completed_count = multiprocessing.Value('i', 0)
    
    processes = []
    for i in range(5):
        p = multiprocessing.Process(target=process_image, args=(i, completed_count))
        processes.append(p)
        p.start()

    for p in processes:
        p.join()

    print(f"Total images processed: {completed_count.value}")

Now it works. It prints "5". But as a software engineer, this immediately makes me nervous. counter.value += 1 is not an atomic operation. It's actually three steps: read the value, add one, write it back. If two processes do the "read" step at the exact same millisecond, they'll both read "10", both increment it to "11", and both write "11" back—and we just lost a count. This is a classic race condition.

Ensuring Data Integrity

To prevent the processes from stepping on each other's toes, I need a Lock. A lock ensures that only one process can modify the shared value at a time. It's like a "talking stick" in a meeting; you can't speak (modify data) unless you're holding the stick.

import multiprocessing
import time

def process_image(image_id, counter, lock):
    print(f"Processing image {image_id}...")
    time.sleep(0.1)
    with lock:
        counter.value += 1

if __name__ == "__main__":
    completed_count = multiprocessing.Value('i', 0)
    lock = multiprocessing.Lock()
    
    processes = []
    for i in range(10):
        p = multiprocessing.Process(target=process_image, args=(i, completed_count, lock))
        processes.append(p)
        p.start()

    for p in processes:
        p.join()

    print(f"Total images processed: {completed_count.value}")

By using with lock:, I'm guaranteeing that the read-modify-write cycle happens uninterrupted. It's a small performance hit, but it's the only way to ensure the numbers are actually correct when you're scaling up to dozens of processes.

When a Single Number Isn't Enough

What if I don't just want a count? What if I want to keep a list of which images failed, or a dictionary mapping image IDs to their processing time? Value and Array are too primitive for that.

This is where the Manager comes in. A Manager creates a server process that holds Python objects and allows other processes to manipulate them using proxies. It's more flexible than shared memory, though slightly slower because it involves inter-process communication (IPC) under the hood.

import multiprocessing

def process_image(image_id, results_dict):
    # Simulating a result: 'Success' or 'Fail'
    status = "Success" if image_id % 2 == 0 else "Fail"
    results_dict[image_id] = status

if __name__ == "__main__":
    with multiprocessing.Manager() as manager:
        # Create a shared dictionary
        shared_results = manager.dict()
        
        processes = []
        for i in range(5):
            p = multiprocessing.Process(target=process_image, args=(i, shared_results))
            processes.append(p)
            p.start()

        for p in processes:
            p.join()
            
        print(f"Final Results: {shared_results}")

The Manager is essentially the "Swiss Army Knife" of shared data. If you need a shared list, shared dict, or even a shared queue, the Manager handles the heavy lifting of syncing that data across the process boundaries for you.

📋 Practical Task

Build a Multi-Process Resource Monitor

You are building a system that monitors the health of several simulated servers. Each "monitor" process should check a server and report back whether the server is 'Online' or 'Offline'.

• Create a multiprocessing.Manager().dict() to store the results, where the key is the server name (e.g., "Server-1") and the value is the status.
• Create a multiprocessing.Value('i', 0) to track the total number of servers checked.
• Spawn 6 processes. Each process should:
  1. Simulate a check using time.sleep(random.random()).
  2. Update the shared dictionary with a random status ('Online' or 'Offline').
  3. Increment the shared counter using a multiprocessing.Lock() to ensure thread-safety.
• Once all processes have joined, print the final dictionary of server statuses and the total count of servers checked.