Memory Optimization
Reducing a Python program's memory footprint means understanding per-object overhead, preferring generators and streaming over materializing everything, and reaching for __slots__ only where it earns its keep.
Memory optimization in Python is about deliberately reducing how much memory a program holds at once, or holds unnecessarily long — not about squeezing every last byte by default. This topic covers the standard techniques (generators, streaming, __slots__, memory profiling) and, just as importantly, when reaching for them is worth the added complexity versus premature.
Object Overhead Revisited
CPython implementation detail: every object carries fixed overhead — at minimum a reference count and a type pointer — before any of its actual data. A user-defined class instance additionally carries, by default, a per-instance __dict__ for its attributes, which itself has real memory cost (a dict has overhead well beyond the sum of its key/value pairs). This overhead is invisible for a handful of objects but adds up directly when you have millions of small instances — a common situation when parsing large datasets into many small record objects.
sys.getsizeof() reports the size of the object itself (CPython-specific), not everything it references; a full accounting of a Point instance's memory footprint includes its __dict__ separately, since that's a distinct object it points to.
import sys
class Point:
def __init__(self, x: float, y: float) -> None:
self.x = x
self.y = y
p = Point(1.0, 2.0)
print(sys.getsizeof(p)) # size of the instance's own header/slots
print(sys.getsizeof(p.__dict__)) # the per-instance __dict__ has its own overhead tooGenerators Instead of Lists
A generator produces values lazily, one at a time, on demand, instead of building an entire collection in memory up front. Where a computation only needs to iterate over a sequence once (not index into it, not iterate it multiple times, not know its length ahead of time), a generator expression or generator function uses O(1) memory regardless of how many items it will eventually produce, versus a list comprehension's O(n).
Dropping the square brackets turns a list comprehension into a generator expression; sum() consumes it lazily, so the full 10 million squared values never coexist in memory — only the running total plus the generator's current position do.
The tradeoff: a generator can only be iterated once (it's exhausted after a full pass) and doesn't support len() or random-access indexing. Use a list when you genuinely need to iterate multiple times, index into the data, or know its length — a generator is for a single, forward pass over data that would otherwise be wastefully materialized.
# builds the entire 10-million-element list in memory before summing
total = sum([n * n for n in range(10_000_000)])
# produces one value at a time; never holds more than a handful in memory
total = sum(n * n for n in range(10_000_000))Streaming Large Data
Streaming means processing data incrementally — a chunk, line, or record at a time — instead of loading an entire file or dataset into memory before processing any of it. This is the natural extension of the generator idea to real I/O: iterating a file object line by line reads only one line into memory at a time, rather than .read()ing the whole file up front.
Iterating a file object directly is line-buffered by the OS/interpreter, not fully materialized — this matters enormously for files far larger than available RAM, where .read() up front simply isn't viable.
from pathlib import Path
# loads the entire (potentially huge) file into memory at once
lines = Path("huge_log.txt").read_text(encoding="utf-8").splitlines()
error_count = sum(1 for line in lines if "ERROR" in line)
# streams: only one line resides in memory at a time
error_count = 0
with open("huge_log.txt", encoding="utf-8") as f:
for line in f:
if "ERROR" in line:
error_count += 1__slots__
By default, each instance of a user-defined class gets a per-instance __dict__ to hold its attributes — flexible (attributes can be added dynamically at any time) but costly in memory when you have many instances. Declaring __slots__ as a class attribute (a tuple/list of allowed attribute names) tells CPython to allocate fixed, dict-free storage for exactly those attributes instead, meaningfully reducing per-instance memory when there are many instances of a small class.
__slots__ trades away dynamic attribute flexibility (and complicates multiple inheritance across classes with different slots) for lower per-instance memory — worthwhile specifically when a class has many, many instances and a small, fixed, known set of attributes.
__slots__ only helps when you have a large number of instances of a small class; for a handful of instances, or a class with a naturally variable attribute set, the flexibility of a normal __dict__-backed class is usually worth more than the memory saved. dataclasses support __slots__ too (@dataclass(slots=True), Python 3.10+).
class PointDict:
def __init__(self, x: float, y: float) -> None:
self.x = x
self.y = y
class PointSlots:
__slots__ = ("x", "y")
def __init__(self, x: float, y: float) -> None:
self.x = x
self.y = y
# PointSlots instances use meaningfully less memory per instance than
# PointDict instances, because they skip the per-instance __dict__ entirely.
# PointSlots also raises AttributeError on any attribute not in __slots__:
p = PointSlots(1.0, 2.0)
# p.z = 3.0 # raises AttributeError -- "z" isn't a declared slotArray-Like Alternatives (Conceptually)
A Python list of numbers stores full Python objects (each with its own object-header overhead) referenced by pointers — flexible (mixed types allowed) but memory-heavy for large numeric data. The standard-library array module stores homogeneous numeric data packed contiguously (like a C array), trading Python-object flexibility per element for much lower memory use when you have a large, uniformly-typed numeric sequence. For serious numeric work, the third-party library numpy goes further, offering packed multi-dimensional arrays plus vectorized operations implemented in compiled code — the standard tool of choice for numeric-heavy workloads, though it's an ecosystem library, not part of the standard library.
array.array requires a type code ("l" for a C long, "d" for double, etc.) up front and only holds that one numeric type, in exchange for storing the values packed together rather than as individually-allocated Python objects referenced by pointers.
import array
# a plain list of a million ints: each element is a full Python int object
numbers_list = list(range(1_000_000))
# array.array("l", ...): packed, homogeneous C-style storage, far less memory
numbers_array = array.array("l", range(1_000_000))Memory Profiling
Guessing at memory usage is as unreliable as guessing at CPU hot spots. The standard-library tracemalloc module traces memory allocations and can report where a program's memory is actually going, snapshot-to-snapshot — covered in depth in the related Performance Tools topic. sys.getsizeof() gives the (CPython-specific) size of a single object, useful for comparing two representations of the same data (as in the __slots__ example above), but doesn't account for objects it references.
Avoid Premature Optimization
Memory-optimization techniques — __slots__, array, hand-rolled streaming — all add complexity or reduce flexibility. Apply them where profiling (or a clear, obvious pattern like 'this loop holds a 10GB list in memory') shows they're actually needed, not reflexively on every class or loop. A __dict__-backed class with a handful of instances, or a list comprehension over a few thousand items, gains nothing from these techniques and only adds friction. Measure, then optimize the part that matters.
Practical Usage
In practice: default to generators/streaming for any single-pass processing of data that could be large, especially file/network I/O; reach for __slots__ specifically when you have a data-holding class instantiated in large numbers (parsed records, graph nodes); use array/numpy for large homogeneous numeric data; and profile with tracemalloc when memory usage is a genuine concern rather than a hunch.
Edge Cases
A generator that's never fully consumed can still hold references (e.g. to whatever it captured in its closure) for as long as the generator object itself is alive — it doesn't free everything just because it's lazy. __slots__ on a subclass doesn't remove a __dict__ inherited from a parent class that doesn't itself declare __slots__ — the parent still grants one, silently undoing the memory saving unless every class in the hierarchy declares __slots__. And remember: names ceasing to reference an object doesn't mean 'automatic deletion at scope exit' — the object becomes collectible once unreachable, and CPython (commonly) reclaims it promptly via reference counting, with a cyclic garbage collector handling reference cycles; other implementations may reclaim it at a different time.
Complete Example
from dataclasses import dataclass
@dataclass(slots=True) # Python 3.10+: dataclass with slots, no per-instance __dict__
class LogRecord:
timestamp: str
level: str
message: str
def parse_log_stream(path: str):
"""Yields one LogRecord at a time -- never holds the whole file in memory."""
with open(path, encoding="utf-8") as f:
for line in f:
timestamp, level, message = line.rstrip("\n").split(" ", 2)
yield LogRecord(timestamp, level, message)
def count_errors(path: str) -> int:
return sum(1 for record in parse_log_stream(path) if record.level == "ERROR")
# even for a multi-gigabyte log file, memory use stays roughly constant:
# one LogRecord (with __slots__, so no per-instance __dict__) exists at a time.Mental Model
Think of a list as buying every item on a shopping list up front and carrying it all home in one trip (peak memory equals everything), while a generator is picking up one item at a time as you need it and setting the previous one down before grabbing the next (memory stays roughly constant). __slots__ is choosing a labeled pillbox with exactly the compartments you need instead of a junk drawer that could hold anything.
Quick Reference
(x for x in ...) - generator expression: O(1) memory, single pass
for line in open(path): - streaming: one line in memory at a time
__slots__ = ("x", "y") - fixed per-instance storage, no __dict__
@dataclass(slots=True) - dataclass + __slots__ (3.10+)
array.array("l", data) - packed homogeneous numeric storage
sys.getsizeof(obj) - CPython-specific: size of obj itself
tracemalloc - where memory is actually being allocatedKey Takeaways
Prefer generators and streaming for single-pass processing of data that could be large — they use roughly constant memory versus a list's memory proportional to size.
__slots__ removes the per-instance __dict__, saving real memory when a class has many instances, at the cost of dynamic-attribute flexibility.
array (and numpy, third-party, for heavier numeric work) packs homogeneous numeric data far more compactly than a list of individually-allocated objects.
Use tracemalloc to find where memory is actually going rather than guessing.
Apply memory-optimization techniques where measurement shows they're needed — reflexive __slots__ or hand-rolled streaming on small, non-hot code adds complexity for no real benefit.
data = [transform(x) for x in huge_source] # materializes everything before use total = sum(data) total = sum(transform(x) for x in huge_source) # generator: constant memory
The list comprehension builds and holds the entire transformed sequence in memory before sum() even starts; the generator expression produces and discards one value at a time, so memory use doesn't grow with the size of huge_source.
class GraphNode: __slots__ = ("value",) class LeafNode(GraphNode): pass # inherits GraphNode's __slots__, but adds no __slots__ of its own leaf = LeafNode(...) leaf.extra = "surprise" # works! LeafNode silently got a __dict__ anyway class GraphNode: __slots__ = ("value",) class LeafNode(GraphNode): __slots__ = () # explicitly declare no additional attributes -- still dict-free leaf = LeafNode(...) # leaf.extra = "surprise" # now correctly raises AttributeError
A subclass that omits __slots__ gets a __dict__ by default, even if its parent declared __slots__ — silently defeating the memory savings. Every class in the hierarchy must declare __slots__ (even as an empty tuple) to keep instances dict-free.
Why does a generator expression typically use much less memory than the equivalent list comprehension?
A list comprehension builds and holds every produced value in memory at once (O(n) memory); a generator expression produces one value at a time on demand and doesn't retain earlier ones once consumed, so its memory use stays roughly constant (O(1)) regardless of how many values it will eventually yield.
What does declaring __slots__ on a class actually change, and what capability does it remove?
It tells CPython to allocate fixed storage for exactly the named attributes instead of giving each instance a per-instance __dict__, reducing memory per instance. It removes the ability to dynamically add attributes not listed in __slots__ — assigning an undeclared attribute raises AttributeError.
Why doesn't __slots__ on a subclass alone guarantee its instances have no __dict__?
If any class in the inheritance chain does not declare __slots__ (including the subclass itself, if it omits it — even implicitly, by not overriding), that class contributes a __dict__ to every instance, silently undoing the memory savings elsewhere in the hierarchy. Every class involved must declare __slots__, even as an empty tuple, for instances to stay dict-free.
Why is streaming a file line-by-line (for line in f:) generally preferable to f.read().splitlines() for a very large file?
Iterating the file object directly reads and holds roughly one line in memory at a time, whereas .read() loads the entire file's contents into memory before any processing can begin — for a file larger than available memory, only the streaming approach is even viable.
What does array.array trade away compared to a plain list, in exchange for lower memory use on numeric data?
It trades flexibility for compactness: array.array only holds one fixed, declared numeric type (a type code like "l" or "d") packed contiguously, unlike a list, which can hold any mix of object types as individually-allocated Python objects referenced by pointers.
Why should __slots__ not be applied reflexively to every class in a codebase?
It only pays off when a class has many instances and a small, fixed, known attribute set; for classes with few instances, or ones that benefit from dynamic attributes, the memory saved is negligible while the loss of flexibility (and complications with multiple inheritance across differently-slotted classes) is a real, ongoing cost — applying it without measurement is premature optimization.
Is it accurate to say a Python object is deleted 'the moment it goes out of scope'? What's the more accurate description?
No — the accurate description is that a name ceasing to reference an object makes that object eligible for collection once it becomes unreachable from any live reference; the implementation then reclaims it per its own memory-management rules. CPython commonly reclaims most unreachable objects promptly via reference counting, with a separate cyclic garbage collector handling reference cycles that refcounting alone can't free, but this exact timing is a CPython implementation detail, not a language guarantee.
Write a generator function read_large_csv_column(path: str, column_index: int) that yields one value from the given column index at a time from a CSV file, without loading the whole file into memory.
def read_large_csv_column(path: str, column_index: int): with open(path, encoding="utf-8") as f: for line in f: fields = line.rstrip("\n").split(",") yield fields[column_index] # for value in read_large_csv_column("huge.csv", 2): # process(value) # only one line's worth of data resides in memory at a time
Because the function is a generator (uses yield) and iterates the open file object line by line rather than calling .read(), memory usage stays roughly constant regardless of the file's total size — each line is parsed, its target column yielded, and then discarded before the next line is read.
What happens on the last line, and why does it differ between the two classes?
e1.bonus = 5000 succeeds because Employee has a normal per-instance __dict__ that accepts any new attribute name dynamically. e2.bonus = 5000 raises AttributeError: 'EmployeeSlots' object has no attribute 'bonus', because __slots__ = ("name", "salary") restricts instances to exactly those two attributes, with no __dict__ to fall back on for anything else.
Write a function total_line_length(path: str) -> int that computes the sum of line lengths in a potentially huge text file using constant memory (never holding more than one line at a time), then compare it in a comment to a version that reads the whole file first.
def total_line_length(path: str) -> int: total = 0 with open(path, encoding="utf-8") as f: for line in f: total += len(line) return total # Contrast with: # def total_line_length_naive(path: str) -> int: # lines = open(path, encoding="utf-8").read().splitlines() # whole file in memory # return sum(len(line) for line in lines) # The naive version's peak memory is proportional to the entire file's size; # the streaming version's peak memory is roughly constant (one line at a time).
Iterating the open file object directly (for line in f:) streams the file lazily, one line at a time, so peak memory usage does not grow with file size — this is the key difference from reading the whole file with .read()/.splitlines() up front, whose memory use is proportional to the file's total size.
class Employee:
def __init__(self, name, salary):
self.name = name
self.salary = salary
class EmployeeSlots:
__slots__ = ("name", "salary")
def __init__(self, name, salary):
self.name = name
self.salary = salary
e1 = Employee("Ada", 90000)
e2 = EmployeeSlots("Ada", 90000)
e1.bonus = 5000
e2.bonus = 5000