Multiprocessing IPC

Separate processes don't share memory, so multiprocessing offers dedicated inter-process communication tools — Queue, Pipe, Manager, shared values, and shared memory — each trading off ease of use against serialization overhead.

Threads within one process share the same memory space; separate processes, by design, do not. Getting data between processes started with the multiprocessing module therefore requires explicit inter-process communication (IPC), and multiprocessing provides several mechanisms for it — each with a different balance of simplicity, safety, and performance.

What Is Multiprocessing IPC and Why It's Needed

When you start a new process (via multiprocessing.Process), the operating system gives it its own separate address space — variables and objects in the parent process are simply not visible to the child, and vice versa, unlike with threading.Thread. Any data that needs to move between processes has to be explicitly serialized (usually pickled), sent through some OS-level channel, and deserialized on the other end. multiprocessing's IPC tools wrap that mechanics behind Python-friendly APIs.

This is a genuine, unavoidable consequence of using separate OS processes — it is not specific to CPython or the GIL, though sidestepping the GIL for CPU-bound parallelism is exactly why processes (rather than threads) are often chosen in the first place.

multiprocessing.Queue

multiprocessing.Queue offers an API similar to queue.Queue (put/get, optional maxsize) but works safely across process boundaries: items are pickled by the sending side and unpickled by the receiving side, moving through an underlying pipe with internal locking to keep multiple producers/consumers safe.

The guard if __name__ == '__main__': is required on platforms that start processes by re-importing the main module (notably Windows, and macOS/Linux when using the 'spawn' start method) — without it, child processes can recursively spawn more children.

import multiprocessing as mp

def worker(q: mp.Queue) -> None:
    q.put("result from child process")

if __name__ == "__main__":
    q: mp.Queue = mp.Queue()
    p = mp.Process(target=worker, args=(q,))
    p.start()
    print(q.get())  # "result from child process"
    p.join()

Pipe

multiprocessing.Pipe() returns a pair of connected Connection objects (by default a duplex/two-way pipe): each end can send() and recv() Python objects (pickled under the hood). A Pipe is a simpler, typically lower-overhead choice than Queue for exactly two communicating endpoints; Queue is the better fit for multiple producers/consumers.

import multiprocessing as mp

def worker(conn: mp.connection.Connection) -> None:
    conn.send({"status": "done", "value": 42})
    conn.close()

if __name__ == "__main__":
    parent_conn, child_conn = mp.Pipe()
    p = mp.Process(target=worker, args=(child_conn,))
    p.start()
    print(parent_conn.recv())  # {'status': 'done', 'value': 42}
    p.join()

Manager

multiprocessing.Manager() starts a separate helper process that holds Python objects (dict, list, and others) and hands out proxy objects to other processes; operations on a proxy are transparently sent to the manager process, which performs them and returns the result — giving the appearance of shared mutable data structures across processes, at the cost of a round trip through the manager for every operation.

Every read or write to a Manager-backed dict/list is an IPC call to the manager process under the hood, which is much slower than a real in-process dict — convenient, but not free.

import multiprocessing as mp

def worker(shared_dict: dict, key: str, value: int) -> None:
    shared_dict[key] = value

if __name__ == "__main__":
    with mp.Manager() as manager:
        shared_dict = manager.dict()
        processes = [
            mp.Process(target=worker, args=(shared_dict, f"key{i}", i))
            for i in range(4)
        ]
        for p in processes:
            p.start()
        for p in processes:
            p.join()
        print(dict(shared_dict))  # {'key0': 0, 'key1': 1, 'key2': 2, 'key3': 3}

Shared Values and Arrays (multiprocessing.Value / Array)

For simple numeric data, multiprocessing.Value and multiprocessing.Array allocate memory in a block genuinely shared between processes (backed by the OS's shared-memory facilities), avoiding pickling entirely for access to that data. They take a ctypes type code (or type) describing the C-level layout, and by default include a lock for safe concurrent access.

get_lock() acquires the Value's built-in lock so the read-modify-write of += is atomic across processes; without it, concurrent increments from multiple processes can race and lose updates.

import multiprocessing as mp

def increment(counter: mp.Value) -> None:
    with counter.get_lock():
        counter.value += 1

if __name__ == "__main__":
    counter = mp.Value("i", 0)  # 'i' = C int, ctypes type code
    processes = [mp.Process(target=increment, args=(counter,)) for _ in range(10)]
    for p in processes:
        p.start()
    for p in processes:
        p.join()
    print(counter.value)  # 10

The shared_memory Module

multiprocessing.shared_memory (added in 3.8) provides lower-level, allocate-a-block-of-raw-memory shared memory (SharedMemory) and a shared list-like structure (ShareableList), useful for sharing large data (like a NumPy array's underlying buffer) between processes without a pickling copy on every access — each process maps the same OS-level shared memory block and reads/writes it directly.

unlink() must be called exactly once (typically by whichever process created the block) to release the OS-level shared memory; every process that attaches to it should call close() when done.

from multiprocessing import shared_memory

# Create a shared memory block and write into it.
shm = shared_memory.SharedMemory(create=True, size=10)
shm.buf[:5] = b"hello"

# In another process, attach by name and read the same bytes:
# existing = shared_memory.SharedMemory(name=shm.name)
# bytes(existing.buf[:5])  -> b'hello'

shm.close()
shm.unlink()  # release the underlying shared memory block

Serialization Overhead

Queue, Pipe, and Manager all move data by pickling it, which costs CPU time and produces a copy — for small messages this is negligible, but for large objects (big lists, NumPy arrays, big dicts) pickling and unpickling can dominate the total time, sometimes erasing the benefit of parallelizing in the first place. Value, Array, and shared_memory avoid this by giving processes direct access to the same underlying memory, at the cost of a lower-level, less Pythonic API and more manual coordination.

Choosing the Right Mechanism

Mechanism         Best for                              Overhead
Queue               many-to-many message passing           pickling per item
Pipe                 exactly two endpoints, simple msgs      pickling per message
Manager               shared mutable dict/list/etc, easy API   IPC round trip per op
Value / Array          small shared numeric state              near-zero (real shared memory)
shared_memory           large shared buffers (arrays, etc.)     near-zero (real shared memory)

Synchronization Alongside IPC

multiprocessing also provides Lock, RLock, Semaphore, Event, and Condition, mirroring threading's primitives but working across processes — needed whenever multiple processes touch shared state (like Value/Array/shared_memory) that isn't itself inherently safe under concurrent access, the way Queue already is internally.

Practical Pitfalls

Forgetting the if __name__ == '__main__': guard causes runaway process creation on platforms using the spawn start method. Passing unpicklable objects (open file handles, database connections, lambdas, most GUI objects) into a Process/Queue/Pipe fails or behaves unexpectedly. Reading a Value/Array without holding its lock during a compound operation (like +=) races under concurrent access. Leaking a SharedMemory block by forgetting unlink() can leave OS resources allocated after your program exits.

Best Practices

Prefer Queue or Pipe for message-passing style coordination; reach for Value/Array/shared_memory only when profiling shows pickling overhead actually matters, since they demand more manual care. Always guard process creation with if __name__ == '__main__':. Always pair shared mutable state with the appropriate lock, and always release shared_memory blocks (close + unlink) explicitly.

Complete Example

import multiprocessing as mp

def producer(q: mp.Queue, count: int) -> None:
    for i in range(count):
        q.put(i * i)
    q.put(None)  # sentinel

def main() -> None:
    q: mp.Queue = mp.Queue()
    proc = mp.Process(target=producer, args=(q, 5))
    proc.start()

    results = []
    while True:
        item = q.get()
        if item is None:
            break
        results.append(item)

    proc.join()
    print(results)  # [0, 1, 4, 9, 16]

if __name__ == "__main__":
    main()

Mental Model

Think of separate processes as separate buildings with no shared filing cabinet: Queue and Pipe are like a courier who photocopies (pickles) a document, drives it to the other building, and hands over the copy; Value, Array, and shared_memory are more like both buildings having a shared, jointly-accessible storage room — faster to use, but you both need to coordinate carefully (locks) so you don't grab the same shelf at the same time.

Quick Reference

multiprocessing.Queue         many-to-many, pickled items
multiprocessing.Pipe()         two-endpoint duplex connection, pickled
multiprocessing.Manager()      proxy objects (dict/list/...), IPC per op
multiprocessing.Value / Array   real shared memory, ctypes types, has a lock
shared_memory.SharedMemory      raw shared buffer, manual attach/close/unlink
Lock / RLock / Semaphore / Event / Condition   cross-process synchronization

Key Takeaways

Separate processes do not share memory, so multiprocessing provides dedicated IPC mechanisms rather than relying on shared variables the way threads can.

Queue, Pipe, and Manager all move data by pickling, which is convenient but adds serialization overhead, especially for large payloads.

Value, Array, and shared_memory give processes direct access to real shared memory, avoiding pickling but requiring manual synchronization and cleanup.

Always guard process-creating code with if __name__ == '__main__': to avoid runaway process spawning on platforms using the spawn start method.

import multiprocessing as mp def worker(): print("hi") p = mp.Process(target=worker) # no __main__ guard p.start() import multiprocessing as mp def worker(): print("hi") if __name__ == "__main__": p = mp.Process(target=worker) p.start() p.join()

On platforms/start methods that re-import the main module to create a child process (Windows always; macOS/Linux with 'spawn'), missing the guard causes the module-level Process() call to run again in the child, spawning children recursively.

counter = mp.Value("i", 0) def increment(): counter.value += 1 # not atomic: read, add, write can race counter = mp.Value("i", 0) def increment(): with counter.get_lock(): counter.value += 1 # atomic under the Value's own lock

+= on a shared Value is a read-modify-write, not a single atomic operation; concurrent processes can interleave and lose updates unless the built-in lock is held for the whole operation.

Why do separate processes need explicit IPC mechanisms, while threads in the same process don't?

Each process gets its own separate address space from the operating system, so variables and objects in one process simply aren't visible in another; threads within one process share that same memory directly, so they can access shared variables without any serialization.

What is the main cost of using multiprocessing.Queue or Pipe to move data between processes?

Every item sent must be pickled by the sender and unpickled by the receiver, which costs CPU time and memory proportional to the object's size — this can become significant for large payloads.

How does multiprocessing.Manager provide 'shared' dicts and lists across processes if processes don't share memory?

It runs a separate helper process that actually holds the real object; other processes get proxy objects, and every operation on a proxy is transparently forwarded as an IPC call to the manager process, which performs it and returns the result.

Why do Value and Array avoid the pickling overhead that Queue, Pipe, and Manager incur?

They allocate a block of genuine OS-level shared memory that multiple processes map directly, so reading or writing them accesses the same underlying bytes rather than copying serialized data through a channel.

Why is if __name__ == '__main__': important when using multiprocessing?

On platforms that create child processes by re-importing the main module (Windows always, and macOS/Linux with the 'spawn' start method), code that creates processes at module level would run again in each child without the guard, causing runaway recursive process creation.

When would you choose multiprocessing.shared_memory over multiprocessing.Value/Array?

For sharing larger blocks of data (like a big array's raw buffer) directly between processes, where Value/Array's simpler ctypes-based model doesn't fit as naturally as a raw shared buffer that can be viewed with something like a memoryview or NumPy array.

Write code that spawns a child process, sends it a number through a Pipe, has the child square it and send the result back, and prints the result in the parent.

import multiprocessing as mp def worker(conn): n = conn.recv() conn.send(n * n) conn.close() if __name__ == "__main__": parent_conn, child_conn = mp.Pipe() p = mp.Process(target=worker, args=(child_conn,)) p.start() parent_conn.send(6) print(parent_conn.recv()) # 36 p.join()

Pipe() gives two connected endpoints; the parent sends a value with send(), the child receives it with recv(), computes, sends the result back, and the parent receives it — each send/recv pickles and unpickles the object across the process boundary.

This code is very likely to print a value less than 400000. Why, and how would you fix it?

counter.value += 1 is a read-modify-write, not atomic; concurrent processes can read the same value before either writes back, losing increments. Fix it by wrapping the increment in with counter.get_lock(): so each increment is atomic across processes.

Explain, in a sentence or two, when you would prefer multiprocessing.Queue over multiprocessing.Value/Array for sharing data between processes.

Prefer Queue when you're passing discrete messages or work items between one or more producers and consumers (a message-passing pattern); prefer Value/Array when you have simple, frequently-updated shared numeric state where the pickling overhead of repeatedly using a Queue would be wasteful.

import multiprocessing as mp

counter = mp.Value("i", 0)

def work():
    for _ in range(100000):
        counter.value += 1

procs = [mp.Process(target=work) for _ in range(4)]
for p in procs: p.start()
for p in procs: p.join()
print(counter.value)