Multiprocessing and Parallel Computing

Threads are several workers in one room sharing the same desk; processes are workers in separate rooms who must pass notes under the door.

Learning Objectives

Choose between threads and processes for parallel work; use process pools; understand fork versus spawn and serialization (pickling) overhead.

Share memory between processes; use pipes and queues; synchronise across processes; initialise workers; manage process lifecycle, child failure and restart; size pools; chunk and batch.

Apply map, reduce and map-reduce; distinguish data parallelism, task parallelism, SIMD, GPU parallelism and distributed computation.

Account for isolation and security, OS differences, Python's GIL context, serialization costs and oversubscription.

Predict speedup with Amdahl's and Gustafson's laws; balance load; reduce correctly; reproduce numerical results; benchmark parallel programs.

Plain-Language Explanation

Threads are several workers in one room sharing the same desk; processes are workers in separate rooms who must pass notes under the door. Separate rooms cost more (each needs its own furniture, and passing notes takes time) but nobody can mess up anyone else's desk — and in some languages (standard CPython), only separate rooms let the workers truly calculate at the same time. Parallel speed-up is never free: the part of the job that cannot be split, and the cost of passing notes, decide how much faster many workers can be.

Precise Technical Explanation

Threads versus processes for parallel work Parallel CPU work in CPython (standard build)

Parallel CPU work in C/C++/Java/Go/Rust/.NET free (shared memory) explicit (serialization, shared memory) ms (spawn: tens to hundreds of ms)

concurrent.futures.ProcessPoolExecutor, multiprocessing.Pool, Node child_process/cluster, ProcessBuilder pools, worker processes in servers (Gunicorn, PHP-FPM).

Fork versus spawn process creation models fork (POSIX): the child is a copy-on-write clone of the parent — fast, inherits state, but unsafe in multithreaded parents (locks held by other threads stay locked in the child).

spawn: start a fresh interpreter and import the main module — slower, clean, the only option on Windows, default on macOS since Python 3.8; Python 3.14 makes forkserver/spawn the default everywhere on POSIX.

forkserver: fork from a clean single-threaded server process.

Serialization and pickling overhead Arguments and results cross process boundaries by serialization (Python pickle, structured clone in browsers, JSON/protobuf elsewhere). Large arguments can cost more than the computation (measured below).

multiprocessing.shared_memory, Value/Array, mmap, POSIX shm_open, Windows file mappings — zero-copy sharing of raw buffers, with process-shared locks for coordination.

Pipes and queues multiprocessing.Pipe (two endpoints), multiprocessing.Queue (pipe + feeder thread + pickling), OS pipes, sockets.

multiprocessing.Lock/Semaphore/Event/Barrier, PTHREAD_PROCESS_SHARED mutexes in shared memory, named semaphores, file locks.

initializer= functions load models, open connections or set up state once per worker instead of once per task.

Start → serve tasks → exit (normal, terminate, crash). Pools join workers at shutdown; zombies must be reaped.

Child process failure A worker that crashes (segfault, os._exit, OOM kill) breaks the pool: Python raises BrokenProcessPool for pending futures (demonstrated below). Robust systems detect failures and replace workers.

maxtasksperchild (multiprocessing.Pool) / max_tasks_per_child (Python 3.11+ executor) recycle workers to contain leaks; supervisors restart crashed workers (Module 26).

CPU-bound: os.cpu_count() (or the process's CPU affinity / container quota). Account for other processes and for nested parallelism inside workers (NumPy BLAS threads) to avoid oversubscription.

Chunking and batching Sending items one by one costs a round-trip each; chunksize sends batches, amortising IPC overhead (but larger chunks balance load worse).

Map, reduce and map-reduce patterns Map independent items in parallel; reduce partial results (sum, merge) — preferably combining per-worker partials rather than synchronising a shared accumulator.

The same operation on many data elements (parallel loops, pool.map, OpenMP parallel for, Java parallel streams, Rayon).

Different operations concurrently (a pipeline, a build graph, independent subsystems).

SIMD versus task parallelism SIMD runs one instruction on multiple lanes within a core (AVX, NEON — 4 to 16 floats per instruction); task parallelism spreads work across cores. They multiply: vectorised code on every core.

GPU parallelism overview Thousands of simple cores executing the same kernel over large data (CUDA, OpenCL, Metal, Vulkan compute, WebGPU) — excellent for regular data-parallel numeric work, poor for branchy or latency-sensitive tasks; data transfer to/from the GPU is often the bottleneck.

Distributed computation versus local multiprocessing Across machines, communication is orders of magnitude slower and partial failures are normal (Module 20). Spark, Dask, Ray, MPI scale out; locally, processes and threads scale up.

Process isolation and security Separate processes can run with different privileges and sandboxes (browser renderer processes, seccomp); a compromised or crashed worker cannot corrupt the parent's memory.

Windows/macOS/Linux differences Windows: spawn only, no fork, handles instead of fds, if __name__ == "__main__": is mandatory. macOS: fork exists but is unsafe with many system frameworks → spawn default. Linux: fork default (changing).

Python multiprocessing and the GIL context In the standard CPython build the GIL allows one thread to execute Python bytecode at a time, so CPU-bound parallelism in pure Python needs processes. Native extensions (NumPy, hashlib on large data, compression) release the GIL and can use threads. Free-threaded builds (Python 3.13+, python3.13t, PEP 703, experimental in 3.13) remove the GIL.

Proportional to data size and complexity; avoid sending large data per task — send indices or file names, use shared memory, or load data in the worker initializer.

Running N processes × M threads each on C cores (N·M ≫ C) thrashes caches and the scheduler. Set inner thread counts (OMP_NUM_THREADS=1) when parallelising at the outer level.

Parallel speedup and Amdahl's law If a fraction *s* of the work is serial, speedup on *N* workers is at most S(N) = 1 / (s + (1 − s)/N); as N → ∞, S → 1/s. With 5% serial work the ceiling is 20×.

Gustafson's law overview If the *problem size* grows with N while serial time stays fixed, scaled speedup is S(N) = N − s·(N − 1) — why big machines are used for bigger problems rather than the same problem faster.

Static partitioning (equal chunks) works for uniform tasks; dynamic scheduling (small chunks from a shared queue, work stealing) for irregular ones.

Parallel reduction correctness Reduction requires an associative operation; floating-point addition is *not* associative, so parallel sums can differ from serial sums in the last bits (shown below).

Reproducible numerical computation Fix the partitioning and reduction order (deterministic trees), use compensated or exact summation (math.fsum, Kahan), fix random seeds per partition, and document tolerances.

Benchmarking parallel programs Report wall time *and* CPU time, scale the worker count (1, 2, 4, 8…), include start-up and serialization, warm up, repeat, check correctness of results, state hardware and settings. Simulated speedups (calculator below) are models, not measurements.

Why It Matters in Real Systems

Data processing, scientific computing, image/video encoding, machine-learning preprocessing and build systems depend on effective parallelism; mis-sized pools, pickling overhead and Amdahl's law explain most disappointing speedups.

Key Terms

Reusable worker processes Clone the parent / start a fresh interpreter Python's object serialization for IPC Data / task parallelism

Same operation on many items / different operations at once Speedup limit from serial fraction Gustafson's law

Scaled speedup with growing problem size More runnable threads than cores Items sent per IPC round-trip

Syntax and API Overview

The specification's process-pool example (CPython 3.13.1):

Expected output [1, 4, 9, 16] — map returns results in input order. The if __name__ == "__main__": guard is required with the *spawn* start method (Windows, macOS default): each worker imports the main module, and without the guard every worker would try to start its own pool. Each call pays process start-up (once per worker) and pickling of arguments and results.

from concurrent.futures import ProcessPoolExecutor

def square(n):
    return n * n

if __name__ == "__main__":
    with ProcessPoolExecutor() as pool:
        print(list(pool.map(square, [1, 2, 3, 4])))

[1, 4, 9, 16]

Complete Code Example

CPU-bound work: sequential versus threads versus processes, chunk size, serialization cost, worker crash detection (CPython 3.13.1 on 8 hardware threads; one real run):

Start methods on Windows versus Linux (the same question asked of Windows CPython 3.13.1 and of WSL's Ubuntu CPython 3.10):

Data parallelism with OpenMP in C (GCC 11.2 -fopenmp -O2), and why parallel floating-point sums differ from sequential ones:

Amdahl's and Gustafson's laws tabulated (a *model*, not a measurement):

import os
import time
from concurrent.futures import ProcessPoolExecutor, ThreadPoolExecutor
from concurrent.futures.process import BrokenProcessPool


def cpu_work(n):
    return sum(i * i for i in range(n))


def echo_size(data):
    return len(data)


def crash(_):
    os._exit(3)                                   # simulate a worker dying (segfault, OOM kill...)


def timed(label, fn):
    t0 = time.perf_counter()
    result = fn()
    print(f"{label:<44} {time.perf_counter() - t0:6.2f} s")
    return result


if __name__ == "__main__":
    jobs = [1_500_000] * 16
    print("cpu_count:", os.cpu_count())
    seq = timed("sequential, 16 CPU tasks", lambda: [cpu_work(n) for n in jobs])
    with ThreadPoolExecutor(8) as tp:
        thr = timed("8 threads (GIL build: no CPU parallelism)", lambda: list(tp.map(cpu_work, jobs)))
    with ProcessPoolExecutor(8) as pp:
        list(pp.map(cpu_work, [10] * 8))          # warm up: start the 8 workers first
        prc = timed("8 processes", lambda: list(pp.map(cpu_work, jobs)))
        assert seq == thr == prc                  # same results all three ways

        tiny = list(range(20_000))
        timed("20,000 tiny tasks, chunksize=1", 
…

cpu_count: 8
sequential, 16 CPU tasks                       2.11 s
8 threads (GIL build: no CPU parallelism)      2.13 s
8 processes                                    0.78 s
20,000 tiny tasks, chunksize=1                 5.23 s
20,000 tiny tasks, chunksize=2,500             0.01 s
send a 50 MB argument to a worker (pickling)   0.13 s
after a worker died: [285, 'BrokenProcessPool', 'BrokenProcessPool']

import multiprocessing as mp
import subprocess

print("Windows CPython 3.13:", mp.get_start_method(), "| available:", mp.get_all_start_methods())
linux = subprocess.run(["wsl", "-e", "python3", "-c",
    "import multiprocessing as m, sys; print(f'Linux CPython {sys.version_info.major}.{sys.version_info.minor}:', m.get_start_method(), '| available:', m.get_all_start_methods())"],
    capture_output=True, text=True)
print(linux.stdout.strip())

Windows CPython 3.13: spawn | available: ['spawn']
Linux CPython 3.10: fork | available: ['fork', 'spawn', 'forkserver']

#include <omp.h>
#include <stdio.h>

int main(void)
{
    const long n = 200000000;
    double t0 = omp_get_wtime();
    double seq = 0.0;
    for (long i = 1; i <= n; ++i) seq += 1.0 / ((double)i * (double)i);
    double t_seq = omp_get_wtime() - t0;

    t0 = omp_get_wtime();
    double par = 0.0;
    #pragma omp parallel for reduction(+:par)        /* each thread sums a partial; partials are combined */
    for (long i = 1; i <= n; ++i) par += 1.0 / ((double)i * (double)i);
    double t_par = omp_get_wtime() - t0;

    printf("threads: %d\n", omp_get_max_threads());
    printf("sequential sum %.17f in %.2f s\n", seq, t_seq);
    printf("parallel   sum %.17f in %.2f s (speedup %.1fx)\n", par, t_par, t_seq / t_par);
    printf("difference: %.3g (floating-point addition is not associative)\n", par - seq);
    return 0;
}

threads: 8
sequential sum 1.64493405783457503 in 0.24 s
parallel   sum 1.64493406183981428 in 0.07 s (speedup 3.6x)
difference: 4.01e-09 (floating-point addition is not associative)

def amdahl(serial, n):
    return 1 / (serial + (1 - serial) / n)

def gustafson(serial, n):
    return n - serial * (n - 1)

print("workers | Amdahl s=5% | Amdahl s=25% | Gustafson s=5%")
for n in [1, 2, 4, 8, 16, 64, 1024]:
    print(f"{n:>7} | {amdahl(0.05, n):11.2f} | {amdahl(0.25, n):12.2f} | {gustafson(0.05, n):14.2f}")
print(f"Amdahl limit as workers -> infinity: s=5% -> {1/0.05:.0f}x, s=25% -> {1/0.25:.0f}x")

workers | Amdahl s=5% | Amdahl s=25% | Gustafson s=5%
      1 |        1.00 |         1.00 |           1.00
      2 |        1.90 |         1.60 |           1.95
      4 |        3.48 |         2.29 |           3.85
      8 |        5.93 |         2.91 |           7.65
     16 |        9.14 |         3.37 |          15.25
     64 |       15.42 |         3.82 |          60.85
   1024 |       19.64 |         3.99 |         972.85
Amdahl limit as workers -> infinity: s=5% -> 20x, s=25% -> 4x

Line-by-Line Explanation

Threads give no speedup for pure-Python CPU work on the GIL build; processes do (limited by core count, SMT and overhead).

The pool is warmed up first so the timing excludes worker start-up (spawn on Windows costs hundreds of ms).

chunksize=1 pays one IPC round-trip per tiny task; chunksize=2500 amortises it.

Sending 50 MB pickles and copies it through a pipe — often more expensive than the computation itself.

os._exit(3) kills a worker; the pool is marked broken and pending futures raise BrokenProcessPool (the first, already finished task keeps its result).

OpenMP splits the loop among threads, each with a private partial sum, and combines them; the result differs in the last digits because the additions happen in a different order.

Output

Shown under each program; timings and the floating-point difference come from one real run.

Interleaving or State Trace

pool.map(cpu_work, 16 jobs) with 8 processes: the parent pickles jobs into the call queue; each worker takes one, computes, pickles the result into the result queue; the parent reorders results by input index. Two waves of 8 → wall time ≈ 2 × one job (plus overhead), versus 16 × one job sequentially.

Correctness and Safety Analysis

Processes share nothing by default, so there are no data races between workers — the correctness concerns move to serialization (only picklable objects), ordering of results (map preserves order, as_completed does not), worker failure, and floating-point reduction order.

Common Mistakes and Misconceptions

Missing if __name__ == "__main__": on Windows/macOS (infinite process spawning or errors).

Passing huge data per task.

Using threads for CPU-bound pure-Python work on the GIL build.

Expecting linear speedup (Amdahl, overhead, SMT, memory bandwidth).

Oversubscribing with nested parallel libraries.

Comparing parallel and serial floating-point results with ==.

Failure Modes and Debugging Steps

BrokenProcessPool: a worker crashed — check native extensions, memory limits (OOM killer), os._exit in libraries; enable faulthandler in workers.

PicklingError: lambdas, open files, sockets and locks cannot be pickled — use module-level functions and pass identifiers.

No speedup: measure where time goes (serialization, start-up, a serial phase, memory bandwidth).

Performance and Scalability Implications

Speedup is bounded by serial fraction, communication, synchronization, load imbalance and shared hardware (memory bandwidth, SMT siblings sharing execution units — 8 hardware threads on 4 cores do not give 8×).

Security Considerations

pickle executes code when loading — never unpickle data from untrusted sources (including untrusted worker processes). Process isolation and least-privilege workers contain compromised components.

Platform and Runtime Caveats

Windows supports only spawn; Linux's default start method is fork in Python ≤ 3.13 (changing to forkserver in 3.14).

Free-threaded CPython 3.13 (python3.13t) is experimental; the standard Windows NuGet build used here has the GIL.

os.cpu_count() reports logical CPUs, not the cores available to a container (Python 3.13 adds os.process_cpu_count()).

OpenMP needs compiler support (-fopenmp GCC/Clang, /openmp MSVC).

Real-World Applications

Python data pipelines (multiprocessing, joblib, Dask), video encoding with ffmpeg threads, scientific codes with OpenMP/MPI, build systems running compilers as processes, web servers with worker processes, ML data loaders.

Best Practices

Use processes for CPU-bound Python, threads for CPU-bound native code and for I/O.

Keep tasks coarse; batch with chunksize; avoid shipping big data.

Initialise expensive state once per worker.

Handle worker failure; recycle workers that leak.

Benchmark with scaling curves; model with Amdahl before buying cores.

Exercises

E1 · Beginner. A program is 90% parallelisable. What is the maximum speedup on 4 cores, and on infinitely many?

Answer: 1 / (0.1 + 0.9/4) ≈ 3.08×; limit 1/0.1 = 10×.

E2 · Intermediate. Why is pool.map(f, items, chunksize=1) slow for 20,000 tiny tasks?

Answer: Each item needs pickling, a pipe write, a worker wake-up, and a result pickle and pipe read — microseconds to milliseconds of IPC for nanoseconds of work. Larger chunks amortise the overhead.

E3 · Advanced challenge. Make the OpenMP sum bit-reproducible regardless of the number of threads.

Answer: Split the range into a fixed number of blocks independent of the thread count (e.g. 1,024), compute each block's partial sum (any thread may compute it), store partials in an array, then add the partials sequentially in block order — the summation order is fixed, so results are identical for any thread count (or use exact/compensated summation).

Quiz

? Why do CPU-bound pure-Python threads not speed up on the standard CPython build?
+ The GIL lets only one thread execute Python bytecode at a time. :: Processes or free-threaded builds are needed.
- Python threads are not real OS threads. :: They are OS threads.
- The OS schedules Python threads on one core. :: The scheduler is not the limit.
- Thread pools are disabled in CPython. :: They work fine for I/O.

? With 5% serial work, what is the Amdahl upper bound on speedup?
+ 20x :: 1 / 0.05.
- 95x :: That ignores the serial part.
- Unlimited :: The serial fraction caps it.
- 5x :: That would be 20% serial.

? Which reduce process-pool overhead? (select all that apply)
+ Larger chunksize for many small tasks :: Fewer IPC round-trips.
+ Loading big data once in a worker initializer :: No per-task transfer.
- Passing a 1 GB list to every task :: Maximises pickling cost.
+ Reusing the pool instead of creating one per batch :: Amortises start-up.

Summary

Processes give isolation and (for CPython) true CPU parallelism at the cost of start-up and serialization; threads share memory cheaply. Fork, spawn and forkserver differ by platform. Chunk work, avoid large transfers, handle worker failure, avoid oversubscription, and model expectations with Amdahl and Gustafson before measuring carefully — remembering that parallel floating-point reductions change results in the last bits.