Shared Memory

multiprocessing.shared_memory lets separate processes access the same block of raw memory directly, avoiding the copying and pickling costs of queues and pipes for large data.

Queues and pipes move data between processes by copying it -- pickling on one end, unpickling on the other. For small messages that's fine, but for large buffers (big arrays, images, matrices) that copying cost can dominate runtime. The multiprocessing.shared_memory module, added in Python 3.8, lets multiple processes map the same underlying block of memory directly, so data can be written once and read by every process without duplication.

What Is Shared Memory

Shared memory is an operating-system feature: a region of memory that multiple, otherwise-isolated processes can each map into their own address space and access directly. Unlike Queue/Pipe, which serialize a copy of the data for transport, shared memory means the processes are literally looking at the same bytes -- writes by one process are visible to others immediately, without any serialization step.

multiprocessing.shared_memory.SharedMemory provides a raw named (or anonymous) block of bytes; multiprocessing.shared_memory.ShareableList provides a simple list-like structure of fixed-size, mutable, picklable-typed values built on top of it. Higher-level tools like multiprocessing.Value/Array (ctypes-backed) are a related, older mechanism for sharing typed scalars/arrays between related processes.

multiprocessing.shared_memory Basics

shm.buf is a memoryview onto the raw shared bytes. shm.name is the OS-level identifier another process uses to attach to the same block.

SharedMemory provides raw bytes with no built-in structure or type information -- you're responsible for interpreting those bytes consistently (e.g. via struct, array, or NumPy's frombuffer) across every process that touches the block.

from multiprocessing import shared_memory

# Create a shared block of 10 bytes
shm = shared_memory.SharedMemory(create=True, size=10)
shm.buf[0] = 65  # write byte 'A' directly into shared memory
print(bytes(shm.buf[:5]))  # b'A\x00\x00\x00\x00'

# Another process can attach to it by name:
existing = shared_memory.SharedMemory(name=shm.name)
print(existing.buf[0])  # 65 -- sees the same underlying memory

existing.close()
shm.close()
shm.unlink()  # release the OS-level shared memory block

Using Shared Memory Across Processes

The child process attaches to the same named block and writes into it as a NumPy array view; because it's the same memory (not a copy), the parent sees those writes after the child finishes, with no queue or pickling involved.

from multiprocessing import Process, shared_memory
import numpy as np

def fill_array(shm_name: str, shape: tuple[int, ...]) -> None:
    shm = shared_memory.SharedMemory(name=shm_name)
    arr = np.ndarray(shape, dtype=np.float64, buffer=shm.buf)
    arr[:] = np.arange(shape[0])
    shm.close()

if __name__ == "__main__":
    shape = (5,)
    nbytes = int(np.prod(shape)) * np.dtype(np.float64).itemsize
    shm = shared_memory.SharedMemory(create=True, size=nbytes)
    arr = np.ndarray(shape, dtype=np.float64, buffer=shm.buf)
    arr[:] = 0

    p = Process(target=fill_array, args=(shm.name, shape))
    p.start()
    p.join()

    print(arr)  # [0. 1. 2. 3. 4.] -- child's writes are visible here

    shm.close()
    shm.unlink()

ShareableList

For simpler cases where you don't need a raw NumPy-style buffer, ShareableList gives a mutable, list-like sequence of a fixed number of items (each item can be an int, float, bool, small str/bytes, or None) backed by shared memory, without needing a separate library to interpret the bytes.

from multiprocessing import shared_memory

sl = shared_memory.ShareableList([1, 2.5, "hi", True, None])
print(sl[2])  # "hi"
sl[0] = 99
print(list(sl))  # [99, 2.5, 'hi', True, None]

# Another process could attach with:
# other = shared_memory.ShareableList(name=sl.shm.name)

sl.shm.close()
sl.shm.unlink()

Synchronization

Shared memory only shares the bytes -- it provides no automatic protection against two processes writing to the same location at the same time. Without coordination, concurrent writes can interleave and corrupt data (a torn write), or a reader can observe a partially-updated value. You must add your own synchronization, typically with a multiprocessing.Lock (or RLock) shared alongside the memory block, guarding any read-modify-write sequence.

Without the lock, concurrent read-modify-write cycles from multiple processes would race and the final count would likely be less than 4000.

from multiprocessing import Process, Lock, shared_memory
import struct

def increment(shm_name: str, lock) -> None:
    shm = shared_memory.SharedMemory(name=shm_name)
    for _ in range(1000):
        with lock:
            current = struct.unpack("i", shm.buf[:4])[0]
            struct.pack_into("i", shm.buf, 0, current + 1)
    shm.close()

if __name__ == "__main__":
    shm = shared_memory.SharedMemory(create=True, size=4)
    struct.pack_into("i", shm.buf, 0, 0)
    lock = Lock()

    processes = [Process(target=increment, args=(shm.name, lock)) for _ in range(4)]
    for p in processes:
        p.start()
    for p in processes:
        p.join()

    print(struct.unpack("i", shm.buf[:4])[0])  # 4000 -- correct only because of the lock
    shm.close()
    shm.unlink()

Ownership and Lifetime

A shared memory block has an OS-level lifetime independent of any single Python process's lifetime -- it is not automatically freed just because the process that created it exits. Every process that attaches to a block should call .close() on its own handle when done (releasing that process's view), and exactly one process -- typically whichever created it -- should call .unlink() exactly once to actually deallocate the underlying OS resource.

Forgetting to .unlink() leaks the shared memory segment at the OS level, where it can persist even after your Python program exits, potentially requiring manual cleanup (e.g. removing /dev/shm/... entries on Linux) or a system restart. Calling .unlink() from more than one process, or using a block after it's unlinked, raises errors -- track ownership carefully, e.g. by having the creator be solely responsible for cleanup, in a try/finally or with-like pattern.

Use Cases

Shared memory is most valuable when: (1) the data being passed between processes is large (big arrays, images, audio buffers) and copying it via a queue would be a measurable bottleneck; (2) multiple worker processes need read access to the same large, mostly-static dataset (loading it once into shared memory instead of once per worker saves both time and total memory); (3) you're building on top of NumPy or similar array libraries, where np.ndarray(..., buffer=shm.buf) gives a zero-copy view directly onto the shared bytes.

For simple message passing of small objects, or when you don't need the raw performance, Queue/Pipe remain simpler and safer choices (see the Multiprocessing topic) -- shared memory trades that simplicity for speed and adds the burden of manual synchronization and lifetime management.

Edge Cases and Pitfalls

Attaching to a SharedMemory block by name after it's been unlinked raises a FileNotFoundError. Resizing is not supported -- a block is a fixed size for its lifetime; if you need more space you must create a new, larger block and migrate the data. On some platforms, shared memory names have length and character restrictions, and names may need to be unique system-wide (not just within your program), so auto-generated names (the default when name is omitted) are usually safer than hardcoded ones in concurrent scenarios.

Practical Usage

A robust pattern: the parent process creates the shared memory block and is solely responsible for .unlink(); it passes the block's .name (a small, easily picklable string) to worker processes via normal arguments; each worker attaches, uses .close() when done, and never calls .unlink(). Wrap creation/cleanup in try/finally to guarantee the block is released even if an exception occurs mid-computation.

Complete Example

from multiprocessing import Process, shared_memory
import numpy as np

def double_in_place(shm_name: str, shape: tuple[int, ...]) -> None:
    shm = shared_memory.SharedMemory(name=shm_name)
    arr = np.ndarray(shape, dtype=np.int64, buffer=shm.buf)
    arr *= 2
    shm.close()

if __name__ == "__main__":
    shape = (1_000_000,)
    nbytes = int(np.prod(shape)) * np.dtype(np.int64).itemsize
    shm = shared_memory.SharedMemory(create=True, size=nbytes)
    try:
        data = np.ndarray(shape, dtype=np.int64, buffer=shm.buf)
        data[:] = np.arange(shape[0])

        p = Process(target=double_in_place, args=(shm.name, shape))
        p.start()
        p.join()

        print(data[:5])  # [0 2 4 6 8] -- doubled in place, no copy needed
    finally:
        shm.close()
        shm.unlink()

Mental Model

Think of shared memory as a shared whiteboard in a hallway rather than sending memos between offices. Anyone with access can write on it or read from it directly, instantly -- no copying a memo, no waiting for delivery. But because everyone shares the same whiteboard, two people writing in the same spot at once creates a mess (a race), so you still need a rule like 'only one person writes at a time' -- that's the lock. And someone has to take responsibility for eventually erasing and removing the whiteboard, or it stays in the hallway forever.

Quick Reference

SharedMemory(create=True, size=n)   - create a raw shared block
SharedMemory(name=existing_name)    - attach to an existing block
shm.buf                              - memoryview onto the raw bytes
shm.close()                          - release this process's handle
shm.unlink()                         - deallocate the OS resource (call once)
ShareableList([...])                 - simple mutable shared sequence
np.ndarray(shape, dtype=..., buffer=shm.buf)  - zero-copy array view
Pair with multiprocessing.Lock() for any concurrent read-modify-write

Key Takeaways

multiprocessing.shared_memory lets separate processes map and access the same physical memory block directly, avoiding queue/pipe serialization costs for large data.

SharedMemory provides raw bytes (interpret with struct/array/NumPy); ShareableList provides a simple typed mutable sequence.

Shared memory provides no built-in synchronization -- concurrent writers need an explicit Lock to avoid data races and torn writes.

Ownership matters: every attaching process should .close() its handle; exactly one process should .unlink() to free the OS resource, and forgetting to do so leaks the segment.

Best suited to large data shared across processes or a mostly-static dataset read by many workers -- for small messages, Queue/Pipe remain simpler.

shm = shared_memory.SharedMemory(create=True, size=1024) #... use it, program ends without cleanup # the OS-level shared memory segment leaks shm = shared_memory.SharedMemory(create=True, size=1024) try:... # use it finally: shm.close() shm.unlink()

Shared memory segments outlive the Python process that created them at the OS level; forgetting to unlink leaks the resource, which can require manual cleanup outside Python.

def worker(shm_name): shm = shared_memory.SharedMemory(name=shm_name) shm.buf[0] += 1 # no lock -- concurrent workers race here shm.close() def worker(shm_name, lock): shm = shared_memory.SharedMemory(name=shm_name) with lock: shm.buf[0] += 1 shm.close()

Shared memory offers no automatic synchronization; concurrent read-modify-write operations from multiple processes need an explicit Lock to avoid lost updates.

unlink() should be called exactly once, by whichever process owns the block's lifetime (typically the creator); calling it from multiple processes, or using the block afterward, raises errors.

How does shared memory differ from communicating via a Queue or Pipe?

A Queue/Pipe serializes (pickles) a copy of the data and sends it to the other process, which unpickles it; shared memory maps the same physical bytes into multiple processes' address spaces, so no copying or serialization is needed -- writes are visible to all attached processes directly.

Why does SharedMemory provide only raw bytes rather than typed Python objects?

Because it's a thin wrapper over an OS-level shared memory block, which is just a buffer; interpreting those bytes as ints, floats, or arrays is left to the caller (via struct, array, or NumPy) since the OS has no concept of Python types.

Why is explicit synchronization still necessary when using shared memory?

Shared memory only shares access to bytes; it does not coordinate concurrent access. Two processes writing to the same location simultaneously (or one reading while another writes) can produce corrupted or inconsistent data, so a Lock is needed around any read-modify-write sequence.

What is the difference between.close() and.unlink() on a SharedMemory object, and who should call each?.close() releases the calling process's own handle/view onto the block and should be called by every process that attached to it;.unlink() deallocates the underlying OS resource entirely and should be called exactly once, typically by the process that created the block.

What happens if a shared memory segment is never unlinked?

It leaks at the OS level -- the memory remains allocated even after the creating Python process exits, potentially requiring manual removal (e.g. of /dev/shm entries on Linux) or persisting until a system restart.

In what scenario does shared memory provide the biggest advantage over queues?

When passing large data (big arrays, images, buffers) between processes, or when multiple workers need read access to the same large, mostly-static dataset -- shared memory avoids the cost of repeatedly copying/serializing that data.

Can you resize a SharedMemory block after creating it?

No -- a SharedMemory block has a fixed size for its lifetime; to hold more data you must create a new, larger block and migrate the contents.

Write code where a parent creates a shared int array, a child process fills it with squares, and the parent reads the result -- using proper cleanup.

from multiprocessing import Process, shared_memory import numpy as np def fill_squares(shm_name: str, shape: tuple[int, ...]) -> None: shm = shared_memory.SharedMemory(name=shm_name) arr = np.ndarray(shape, dtype=np.int64, buffer=shm.buf) arr[:] = np.arange(shape[0]) ** 2 shm.close() if __name__ == "__main__": shape = (6,) nbytes = int(np.prod(shape)) * np.dtype(np.int64).itemsize shm = shared_memory.SharedMemory(create=True, size=nbytes) try: p = Process(target=fill_squares, args=(shm.name, shape)) p.start() p.join() result = np.ndarray(shape, dtype=np.int64, buffer=shm.buf) print(list(result)) # [0, 1, 4, 9, 16, 25] finally: shm.close() shm.unlink()

The parent creates the block and passes only its name (a small string) to the child; the child attaches, computes squares directly into the shared bytes, and exits; the parent then reads the same memory and sees the child's writes with no copying involved.

What happens when this code runs?

It raises a FileNotFoundError (or similar OS-level error) on the last line -- once unlink() has been called, the OS-level shared memory segment is deallocated, so attempting to attach to it by name afterward fails.

Write code demonstrating why a Lock is needed when multiple processes increment a shared counter stored in shared memory, and show the fix.

from multiprocessing import Process, Lock, shared_memory import struct def increment_unsafe(shm_name: str) -> None: shm = shared_memory.SharedMemory(name=shm_name) for _ in range(2000): current = struct.unpack("i", shm.buf[:4])[0] struct.pack_into("i", shm.buf, 0, current + 1) shm.close() def increment_safe(shm_name: str, lock) -> None: shm = shared_memory.SharedMemory(name=shm_name) for _ in range(2000): with lock: current = struct.unpack("i", shm.buf[:4])[0] struct.pack_into("i", shm.buf, 0, current + 1) shm.close() if __name__ == "__main__": shm = shared_memory.SharedMemory(create=True, size=4) struct.pack_into("i", shm.buf, 0, 0) lock = Lock() processes = [Process(target=increment_safe, args=(shm.name, lock)) for _ in range(3)] for p in processes: p.start() for p in processes: p.join() print(struct.unpack("i", shm.buf[:4])[0]) # 6000, reliably shm.close() shm.unlink() increment_unsafe would race: two processes can read the same current value before either writes back, losing updates, so the total often ends up less than 6000. increment_safe wraps the read-modify-write in a shared Lock, ensuring only one process performs the sequence at a time, so the final count is correct.

from multiprocessing import shared_memory

shm = shared_memory.SharedMemory(create=True, size=4)
shm.close()
shm.unlink()

other = shared_memory.SharedMemory(name=shm.name)