I see this mistake all the time when developers start scaling their Python apps. They have a huge dataset—maybe a 500MB list of coordinates or a massive configuration dictionary—and they pass it directly into the Process constructor. They assume that because they're just passing a reference, the child process is looking at the same memory. It feels efficient, and it seems like it should work.

Passing a list isn't actually sharing memory

Here is the cold hard truth: when you pass a Python object to a child process, Python pickles that object and sends it over a pipe to the new process. The child process then unpickles it. You aren't sharing a list; you're creating a complete, independent copy of that list in a different memory address space. If the child process modifies the list, the parent sees absolutely nothing.

import multiprocessing

def update_data(data):
    data.append("New Item")
    print(f"Child sees: {data}")

if __name__ == "__main__":
    my_list = ["Original Item"]
    p = multiprocessing.Process(target=update_data, args=(my_list,))
    p.start()
    p.join()
    print(f"Parent sees: {my_list}") 
    # Output: Parent sees: ['Original Item']

If your dataset is 1GB and you spawn four workers this way, you've suddenly eaten 5GB of RAM. That's how you get the dreaded OutOfMemoryError on a machine that seemed to have plenty of headroom.

Mapping raw bytes with SharedMemory

To actually share memory, we need to move outside the realm of standard Python objects and deal with raw bytes. This is where multiprocessing.shared_memory comes in. Instead of copying an object, it allocates a block of memory that is identified by a unique name. Any process that knows that name can "attach" to that same physical slice of RAM.

I usually think of it like a shared whiteboard in an office. Instead of photocopying the whiteboard and handing a copy to every employee (which is what pickling does), everyone just walks up to the same board and reads or writes to it.

from multiprocessing import shared_memory

# Process A: Create the memory block
shm = shared_memory.SharedMemory(create=True, size=1024, name="my_shared_block")
buffer = shm.buf # This is a memoryview
buffer[:4] = b"test"

# Process B: Attach to the existing block
existing_shm = shared_memory.SharedMemory(name="my_shared_block")
print(bytes(existing_shm.buf[:4])) # Output: b'test'

# Clean up
existing_shm.close()
shm.close()
shm.unlink() # Only one process should ever call unlink()

Making raw bytes useful with NumPy

Working with shm.buf is tedious because you're dealing with bytes. In a professional setting, you'll almost always wrap this shared memory in a NumPy array. This gives you the speed of shared memory with the API of a standard array. You essentially tell NumPy: "Don't allocate new memory; just use this existing block of bytes as your storage."

import numpy as np
from multiprocessing import shared_memory

# Create shared memory for an array of 1000 doubles (8 bytes each)
shm = shared_memory.SharedMemory(create=True, size=1000 * 8)
# Create a NumPy array backed by that shared memory
shared_array = np.ndarray((1000,), dtype=np.float64, buffer=shm.buf)

shared_array[:] = np.random.random(1000)
print(shared_array[0]) # This is now accessible by any process that attaches to shm.name

The danger of orphaned memory blocks

Here is a warning: shared memory exists independently of your Python script's lifecycle. If your program crashes before you call shm.unlink(), that memory stays allocated. I've seen servers slowly bleed RAM over a week because a developer forgot to unlink their shared blocks after a crash. Always wrap your cleanup in a try...finally block or use a manager to ensure that unlink() is called when the work is done.

📋 Practical Task

Build a Shared Memory Image Brightener

You need to build a system that simulates brightening a grayscale image (represented as a large array of integers) using multiple processes. Because the "image" is large, you must use shared_memory to avoid copying the data for every worker.

Requirements:

• Create a shared memory block that can hold a NumPy array of 1,000,000 integers (use np.int32).
• Initialize this array with random values between 0 and 100.
• Spawn two worker processes. Each worker should be responsible for one half of the array.
• The workers should add 50 to every value in their assigned slice of the shared array.
• The main process must wait for workers to finish, verify that the values have increased, and then properly close() and unlink() the shared memory.

Tip: Remember that the child processes need to create their own SharedMemory instance using the name of the block created by the parent before they can create the NumPy wrapper.