Multiprocessing

The multiprocessing module runs separate Python processes, each with its own interpreter and memory space, to get real parallelism for CPU-bound work.

The multiprocessing standard-library module lets a Python program spawn additional OS processes and coordinate work between them. Unlike threads, which share one process's memory and one interpreter, each process created by multiprocessing runs its own independent Python interpreter with its own memory space — so it sidesteps the GIL entirely (see the related GIL & CPython topic) and can use multiple CPU cores at once for genuine parallelism.

What Is Multiprocessing

multiprocessing is a standard-library package for spawning processes with an API deliberately modeled on the threading module, so code that already understands Thread objects can largely reuse the same mental model for Process objects. The key difference is isolation: each process gets its own copy of the Python interpreter, its own address space, and its own copy of the objects that existed when the process was created.

Because each process is a separate OS-level process (not a lightweight thread inside one process), processes do not share memory by default. Anything one process needs from another must be explicitly sent across a channel — a queue, a pipe, or an explicit shared-memory region — rather than simply read from a shared variable.

Process and Process Objects

The core building block is the Process class. You construct one with a target callable (and optional args/kwargs), then call .start() to launch it and .join() to wait for it to finish. .start() returns immediately; the new process runs concurrently with the parent.

Each Process runs square in its own interpreter. .join() blocks the parent until that child process exits.

On Windows and on macOS (default since Python 3.8), new processes are created with the spawn start method, which re-imports your module in the child — this is why top-level process-creation code must be guarded by if __name__ == "__main__":. On Linux, the default start method is fork, which copies the parent process's memory, so the guard is less strictly required there but is still good practice for portability.

import multiprocessing as mp

def square(n: int) -> None:
    print(f"{n} squared is {n * n}")

if __name__ == "__main__":
    processes = [mp.Process(target=square, args=(i,)) for i in range(4)]
    for p in processes:
        p.start()
    for p in processes:
        p.join()
    print("all workers finished")

Process Isolation

Process isolation means a child process cannot see or mutate the parent's Python objects after it starts (with fork, it inherits a snapshot at creation time; with spawn/forkserver, it starts fresh and only receives what you explicitly pass). This is the opposite tradeoff from threads: isolation buys you safety from race conditions on shared Python objects and true multi-core parallelism, at the cost of needing explicit communication and higher per-process overhead (memory footprint, interpreter startup time, serialization for message passing).

The child mutates its own copy of counter; the parent's counter is untouched because the two live in separate address spaces.

import multiprocessing as mp

counter = 0

def increment():
    global counter
    counter += 1
    print(f"inside child: counter = {counter}")

if __name__ == "__main__":
    p = mp.Process(target=increment)
    p.start()
    p.join()
    print(f"in parent: counter = {counter}")  # still 0 -- child had its own copy

IPC Concept

Since processes don't share memory, they need Inter-Process Communication (IPC) — explicit mechanisms for moving data between them. multiprocessing provides several: Queue and Pipe for message passing (data is pickled, sent, and unpickled on the other end), and shared_memory / shared Value/Array for actually sharing a memory region (see the dedicated Shared Memory topic for depth). Choosing between them is a tradeoff between simplicity (queues/pipes) and raw throughput (shared memory avoids serialization).

Queues

multiprocessing.Queue is a process-safe FIFO queue: multiple processes can .put() and .get() items without corrupting each other's data, because the queue handles the necessary locking and serialization internally. It's the most common way to send results from worker processes back to a parent, or to distribute work items to a pool of workers.

import multiprocessing as mp

def worker(q: mp.Queue, n: int) -> None:
    q.put(n * n)

if __name__ == "__main__":
    q: mp.Queue = mp.Queue()
    processes = [mp.Process(target=worker, args=(q, i)) for i in range(5)]
    for p in processes:
        p.start()
    for p in processes:
        p.join()

    results = [q.get() for _ in range(5)]
    print(sorted(results))  # [0, 1, 4, 9, 16]

Pipes

multiprocessing.Pipe() returns a pair of connected Connection objects (like two ends of a duplex phone line) for point-to-point communication between exactly two processes. It's lighter weight than a Queue when you know you only have two endpoints, and supports .send()/.recv() for arbitrary picklable objects.

A Pipe is unbuffered relative to a Queue and offers no built-in protection against corruption if more than two processes read/write the same end concurrently — use Queue when more than two participants are involved.

import multiprocessing as mp

def worker(conn: mp.connection.Connection) -> None:
    conn.send("hello from child")
    conn.close()

if __name__ == "__main__":
    parent_conn, child_conn = mp.Pipe()
    p = mp.Process(target=worker, args=(child_conn,))
    p.start()
    print(parent_conn.recv())  # "hello from child"
    p.join()

Shared Memory Overview

For cases where copying data through a queue is too slow (e.g. large NumPy-style buffers), multiprocessing offers ways to share an actual memory region instead of message-passing copies: Value and Array (typed, ctypes-backed shared objects, often paired with a Lock), and the multiprocessing.shared_memory module (raw shared memory blocks, Python 3.8+). This is covered in depth in the dedicated Shared Memory topic — the concept to take away here is that shared memory trades the safety of isolation for speed, so it requires explicit synchronization to avoid races.

import multiprocessing as mp

def worker(shared_count, lock):
    with lock:
        shared_count.value += 1

if __name__ == "__main__":
    count = mp.Value("i", 0)  # shared C int
    lock = mp.Lock()
    processes = [mp.Process(target=worker, args=(count, lock)) for _ in range(10)]
    for p in processes:
        p.start()
    for p in processes:
        p.join()
    print(count.value)  # 10

Practical Usage

Reach for multiprocessing when work is CPU-bound (heavy computation: image processing, numerical simulation, parsing large files) and you have multiple cores to exploit. For most everyday parallel-worker code, prefer the higher-level concurrent.futures.ProcessPoolExecutor (see the Process Pools topic) over managing raw Process objects by hand — it gives you a pool, automatic result collection, and cleaner shutdown with far less boilerplate.

Edge Cases and Pitfalls

Everything sent between processes (function arguments, queue/pipe items) must be picklable — lambdas, open file handles, database connections, and local closures generally cannot cross process boundaries. Exceptions raised in a child process are not automatically re-raised in the parent; you must check exit codes or explicitly send error information back through a queue. Process creation itself has real overhead (tens of milliseconds), so spawning a fresh process per tiny task is often slower than doing the work sequentially — pools amortize this cost across many tasks.

Complete Example

import multiprocessing as mp

def cpu_bound_task(n: int) -> int:
    total = 0
    for i in range(n):
        total += i * i
    return total

def worker(n: int, q: mp.Queue) -> None:
    q.put((n, cpu_bound_task(n)))

if __name__ == "__main__":
    inputs = [10_000_000, 12_000_000, 8_000_000, 15_000_000]
    q: mp.Queue = mp.Queue()
    processes = [mp.Process(target=worker, args=(n, q)) for n in inputs]

    for p in processes:
        p.start()
    results = [q.get() for _ in processes]
    for p in processes:
        p.join()

    for n, total in sorted(results):
        print(f"sum of squares below {n}: {total}")

Mental Model

Think of threads as roommates sharing one apartment (one process's memory) and processes as neighbors in separate apartments (separate memory). Roommates can grab the same fridge item directly but might collide doing so; neighbors never collide by accident, but if one needs something from the other's fridge, they have to hand it across the fence explicitly — that hand-off is IPC.

Quick Reference

Process(target=fn, args=(...)) - create a child process
p.start() / p.join()          - launch / wait for completion
Queue()                        - process-safe FIFO for message passing
Pipe()                         - two-endpoint duplex connection
Value(typecode, val), Array()  - shared ctypes-backed memory
Lock() / RLock()               - mutual exclusion across processes
if __name__ == "__main__":     - required guard for spawn/forkserver

Key Takeaways

multiprocessing runs separate OS processes, each with its own interpreter and memory, giving true multi-core parallelism unavailable to threads under CPython's GIL.

Processes are isolated by default — data must be explicitly moved between them via Queue, Pipe, or shared memory.

Guard process-creation code with if __name__ == "__main__": for correctness across start methods (spawn, fork, forkserver).

Everything crossing a queue or pipe must be picklable; process creation has real overhead, so batch work rather than spawning per tiny task.

For everyday parallel workloads, prefer the higher-level ProcessPoolExecutor over raw Process management.

import multiprocessing as mp def worker(): print("hi") p = mp.Process(target=worker) p.start() # on Windows/macOS spawn: may recursively re-import and re-run this module import multiprocessing as mp def worker(): print("hi") if __name__ == "__main__": p = mp.Process(target=worker) p.start() p.join()

Without the __name__ == "__main__" guard, the spawn start method re-imports the module in each child, which can re-trigger process creation and cause infinite recursion or duplicated work.

counter = 0 def worker(): global counter counter += 1 # mutates only the child's private copy p = mp.Process(target=worker) p.start(); p.join() print(counter) # still 0 -- surprises many newcomers count = mp.Value("i", 0) def worker(shared): with shared.get_lock(): shared.value += 1 p = mp.Process(target=worker, args=(count,)) p.start(); p.join() print(count.value) # 1

Ordinary Python objects are not shared between processes; use multiprocessing.Value/Array/shared_memory when a child genuinely needs to update state the parent will observe.

p = mp.Process(target=lambda: print("hi")) p.start() # PicklingError: lambdas can't be pickled for spawn def say_hi(): print("hi") p = mp.Process(target=say_hi) p.start()

Targets and arguments sent to a new process must be picklable; lambdas and many closures are not, so define ordinary named functions at module scope instead.

Why can't a child process modify a module-level variable and have the parent see the change?

Each process gets its own separate memory space and, under spawn, its own fresh interpreter; a child's variables are independent copies (or fresh bindings), not shared storage, so mutations in the child never propagate back to the parent without explicit IPC.

What is the purpose of the if __name__ == "__main__": guard in multiprocessing code?

It prevents process-creation code from re-running when a child process re-imports the module (as happens under the spawn and forkserver start methods), avoiding infinite recursive process creation.

When would you choose a Pipe over a Queue?

When you have exactly two communicating processes and want lower overhead point-to-point communication; Queue is preferred for multiple producers/consumers because it manages concurrent access safely.

Why must arguments passed to a Process target be picklable?

Because (except under fork, which copies memory directly) the arguments must be serialized to be transmitted to the new process's separate memory space and then deserialized there; objects like open sockets, lambdas, and some closures cannot be pickled.

What does process isolation buy you compared to threads, and what does it cost?

It buys freedom from data races on shared Python objects and true multi-core parallelism (bypassing the GIL); it costs the need for explicit communication (IPC), higher memory use, and slower process startup compared to threads.

How do you wait for a spawned process to finish before continuing?

Call .join() on the Process object; it blocks the calling process until the target process terminates.

Why might spawning one Process per tiny task be slower than doing the work sequentially?

Process creation carries real overhead (interpreter startup, memory allocation) on the order of tens of milliseconds, which can dwarf the time saved on a very small task.

Write a function that spawns 3 processes, each computing the cube of a different number, and collects all results via a Queue.

import multiprocessing as mp def cube_worker(n: int, q: mp.Queue) -> None: q.put(n ** 3) def run_cubes(numbers: list[int]) -> list[int]: q: mp.Queue = mp.Queue() processes = [mp.Process(target=cube_worker, args=(n, q)) for n in numbers] for p in processes: p.start() results = [q.get() for _ in processes] for p in processes: p.join() return results if __name__ == "__main__": print(sorted(run_cubes([1, 2, 3]))) # [1, 8, 27]

Each process computes one cube and puts it on the shared Queue; the parent drains the queue once results are available, then joins all processes to ensure clean shutdown.

What does this print, and why?

It prints [1, 2, 3]. The child process operates on its own copy of the list x; appending to it in the child has no effect on the parent's list because the two processes do not share memory.

Write code demonstrating two-way communication between a parent and one child process using Pipe.

import multiprocessing as mp def worker(conn): msg = conn.recv() conn.send(msg.upper()) conn.close() if __name__ == "__main__": parent_conn, child_conn = mp.Pipe() p = mp.Process(target=worker, args=(child_conn,)) p.start() parent_conn.send("hello") print(parent_conn.recv()) # "HELLO" p.join()

The parent sends a string down its end of the pipe; the child receives it, transforms it, and sends the result back; the parent reads the reply. Pipe endpoints support both send and recv, making two-way communication straightforward.

import multiprocessing as mp

x = [1, 2, 3]

def append_item():
    x.append(4)

if __name__ == "__main__":
    p = mp.Process(target=append_item)
    p.start()
    p.join()
    print(x)