Memory Management and Garbage Collection in Runtimes

Learning Objectives

Explain allocation, managed heaps, reachability and roots; reference counting and cycle detection; tracing collectors (mark-sweep, mark-compact, copying); generational, incremental and concurrent collection.

Describe stop-the-world pauses, write and read barriers, remembered sets, allocation fast paths and thread-local allocation buffers.

Handle finalisation, weak references, deterministic destruction, resource cleanup and memory leaks in GC'd languages; read heap snapshots and GC logs; tune trade-offs including real-time needs.

Compare CPython's reference counting with tracing collectors, manual management and ownership (Rust); understand memory pressure and out-of-memory behaviour.

Prerequisites

Modules 7 and 11; the Memory & Program Execution course helps.

Plain-Language Explanation

Programs keep creating objects. A garbage collector finds the objects no running code can reach any more and reuses their memory. It starts from the roots — variables currently in use, globals — follows every reference, and anything it never reaches is garbage. Different collectors trade pause length, throughput and memory overhead in different ways.

Precise Technical Explanation

Runtime object allocation new, object literals, closures, boxed values and strings allocate memory on the managed heap; the runtime chooses where.

Managed heap concept A region (or many regions) owned by the runtime; programs never free memory explicitly.

An object is live if it is reachable from a root through references; liveness by reachability is conservative (a reachable object may never be used again).

Roots: stacks, globals, registers, handles (implementation-dependent)

Local variables in frames, operand-stack values, globals and statics, CPU registers in JIT-compiled code, and handles held by native code. Finding roots precisely needs stack maps from the compiler; conservative collectors treat any pointer-like word as a root.

Each object counts references to it; when the count drops to zero it is freed immediately. Deterministic and incremental, but costs a write on every reference change and cannot reclaim cycles alone.

CPython adds a cycle collector for container objects: it subtracts internal references within a candidate set and frees groups whose counts are explained entirely by internal references.

Tracing garbage collection Periodically traces from roots and reclaims everything unreached.

Mark reachable objects (a bit per object), then sweep the heap freeing unmarked objects into free lists. Simple; can fragment memory.

Mark, then slide live objects together to eliminate fragmentation and update references.

Copy reachable objects from "from-space" to "to-space" (Cheney's algorithm), which compacts as a side effect; cost proportional to live data; needs spare space.

Most objects die young (the weak generational hypothesis): collect the young generation often and cheaply, promote survivors to an old generation collected rarely.

Young and old generations Young objects are allocated in a nursery/eden; survivors move to survivor spaces and then to the old generation.

Minor and major collection terminology (runtime-specific) Minor (young) versus major/full (whole heap) collections;.NET numbers generations 0, 1, 2; HotSpot logs "Pause Young" and "Pause Full".

Collector threads work while the program runs (G1's concurrent marking, ZGC, Shenandoah, .NET background GC), needing barriers to stay correct.

Work is split into small steps interleaved with the program (Lua's incremental collector, V8's incremental marking).

Phases where all program threads stop (at safepoints) so the collector sees a consistent heap.

Write barriers and read barriers Code the runtime runs on reference writes (or reads) to tell the collector about changes — needed for generational remembered sets and concurrent marking.

Records of old-to-young references, so a minor collection can treat them as roots without scanning the whole old generation (card tables in HotSpot).

Allocation fast paths Bump-pointer allocation: if top + size <= end, the object goes at top and top += size — a few instructions.

Thread-local allocation buffers Each thread gets a private chunk (TLAB) for bump allocation without synchronisation.

Finalizers run late, in unspecified order, on another thread, possibly never; they can resurrect objects. Java deprecated finalize (JEP 421); use cleaners or explicit close.

References that do not keep an object alive (weakref, WeakReference, WeakRef); used for caches and observers.

Soft/phantom/finalization reference concepts where applicable

Java's soft references (cleared under memory pressure), phantom references (post-mortem cleanup via Cleaner); JavaScript's FinalizationRegistry.

Deterministic destruction versus garbage collection RAII (C++), reference counting (CPython, Swift ARC) and ownership (Rust) free at predictable points; tracing GC frees "eventually".

Resource cleanup: files, sockets, locks, native memory Garbage collection reclaims unreachable *managed objects*, but it does not guarantee immediate cleanup of operating-system resources. Close them deterministically: Python with, Java try-with-resources, C# using, JavaScript try/finally (and using declarations where supported).

Memory leaks in garbage-collected languages Objects kept reachable by mistake: growing caches and maps, listeners never removed, closures capturing large objects, static collections, thread-locals.

Retained object graphs A small object can retain a huge graph (a cache entry holding a request holding a response buffer); heap tools show retained size.

Heap snapshots and GC logs Snapshots list objects and reference paths (Chrome DevTools, jcmd GC.heap_dump, dotnet-gcdump); GC logs record each collection's type, pause and heap sizes (-Xlog:gc, --trace-gc).

GC tuning trade-offs Throughput (time not spent in GC), latency (pause times) and footprint (heap size): improve two and the third usually suffers.

Real-time and low-latency considerations Bounded-pause collectors (ZGC, Shenandoah, Metronome), avoiding allocation in hot paths, object pools, region-based allocation.

Reference counting in CPython versus tracing collectors in other runtimes

CPython frees most objects immediately when their count drops to zero, plus a generational cycle collector for containers; the JVM, CLR and JavaScript engines use tracing (mostly generational, moving) collectors with no reference counts.

Manual memory management versus managed memory C/C++ malloc/free: full control, risk of leaks, double frees and use-after-free. Managed memory removes those bugs at the cost of runtime overhead and less predictable timing.

Ownership-based approaches such as Rust The compiler tracks a single owner and borrows; memory is freed when the owner goes out of scope — deterministic and without a collector; Rc/Arc add reference counting when sharing is needed.

Memory pressure and out-of-memory behavior When the heap cannot grow, runtimes collect aggressively and then fail: java.lang.OutOfMemoryError, Python MemoryError, Node.js aborts with "JavaScript heap out of memory"; containers may kill the process first.

Why It Matters in Real Systems

GC pauses show up as latency spikes; leaks in managed languages cause gradual slowdowns and crashes; resource leaks exhaust file handles and connections regardless of GC.

Key Terms

Reference the GC starts tracing from Connected to a root by references Collect young objects often, old rarely

Code run on reference stores for the collector Thread-local allocation buffer

Reference that does not keep its target alive

Syntax or Instruction Format

No dedicated source syntax; this is a runtime or architecture concept. Collector APIs and flags: Python gc, sys.getrefcount, weakref; Java -Xlog:gc, -XX:+UseSerialGC; .NET GC.CollectionCount; Node --trace-gc.

Complete Example

CPython: reference counts free objects immediately, a cycle needs the cycle collector, and weak references do not keep objects alive (CPython 3.13.1):

A tracing, generational collector at work — HotSpot's serial collector with GC logging (Temurin JDK 21, 64 MB maximum heap; pause times are one real run):

Output (each Pause Young line is one minor collection: heap used before→after(capacity) and pause time):

Deterministic cleanup does not depend on GC (CPython 3.13.1):

import gc, sys, weakref


class Node:
    def __init__(self, name):
        self.name = name
        self.other = None

    def __del__(self):
        print(f"  freed {self.name}")


a = Node("a")
print("refcount of a (includes the getrefcount argument):", sys.getrefcount(a))
alias = a
print("after alias:", sys.getrefcount(a))
del alias
print("deleting the last reference frees immediately:")
del a

gc.disable()                      # show that plain reference counting cannot free a cycle
x, y = Node("x"), Node("y")
x.other, y.other = y, x           # x <-> y
w = weakref.ref(x)
del x, y
print("cycle unreachable but not freed yet; weakref alive?", w() is not None)
print("cycle collector found", gc.collect(), "unreachable objects:")
print("weakref alive after collection?", w() is not None)
gc.enable()

refcount of a (includes the getrefcount argument): 2
after alias: 3
deleting the last reference frees immediately:
  freed a
cycle unreachable but not freed yet; weakref alive? True
  freed x
  freed y
cycle collector found 9 unreachable objects:
weakref alive after collection? False

public class Alloc {
    public static void main(String[] args) {
        long total = 0;
        for (int i = 0; i < 150_000; i++) {
            byte[] garbage = new byte[1024];        // dies immediately: young-generation garbage
            total += garbage.length;
        }
        System.out.println("allocated about " + total / (1024 * 1024) + " MB of short-lived arrays");
    }
}

[0.009s][info][gc] Using Serial
[0.545s][info][gc] GC(0) Pause Young (Allocation Failure) 17M->4M(61M) 10.155ms
[0.579s][info][gc] GC(1) Pause Young (Allocation Failure) 21M->4M(61M) 5.401ms
[0.584s][info][gc] GC(2) Pause Young (Allocation Failure) 21M->4M(61M) 1.402ms
[0.588s][info][gc] GC(3) Pause Young (Allocation Failure) 21M->4M(61M) 0.656ms
[0.591s][info][gc] GC(4) Pause Young (Allocation Failure) 21M->4M(61M) 0.599ms
[0.595s][info][gc] GC(5) Pause Young (Allocation Failure) 21M->4M(61M) 0.616ms
[0.598s][info][gc] GC(6) Pause Young (Allocation Failure) 21M->4M(61M) 0.598ms
allocated about 146 MB of short-lived arrays

import os, tempfile

path = os.path.join(tempfile.mkdtemp(), "data.txt")
with open(path, "w") as f:          # the file is closed when the block ends, error or not
    f.write("hello")
print("closed right after the with-block:", f.closed)

closed right after the with-block: True

Line-by-Line Explanation

sys.getrefcount reports one extra reference (its own argument); del a drops the count to zero and __del__ runs immediately — CPython-specific behaviour.

The x ↔ y cycle keeps both counts at 1 after del, so reference counting alone never frees them; gc.collect() finds them (it reports the number of unreachable objects, which includes their attribute dictionaries) and runs their finalisers; the weak reference then reports the object gone.

JVM log: the young generation fills (~17 MB) and each minor GC copies the few survivors and frees the rest in a few milliseconds or less; the first pause is the longest. The program's own garbage is all short-lived.

with closes the file deterministically, independent of when the file object itself is collected.

Expected Output

As shown; the number of GCs and pause times vary between runs and versions.

Step-by-Step Runtime Behavior

Minor GC: stop the threads → trace from roots plus remembered-set entries into the young generation → copy survivors → reset eden's bump pointer → resume.

Common Misconceptions

"GC prevents memory leaks." It prevents leaks of *unreachable* objects; reachable-but-unused objects leak.

"Finalizers are destructors." They are unreliable; use explicit cleanup.

"Reference counting has no pauses." Freeing a large structure can cascade; cycles need a separate collector.

Common Bugs and Failure Modes

Unbounded caches, forgotten listeners, closures holding large objects, file/socket leaks relying on GC, OutOfMemoryError from heaps sized below the working set, GC thrashing near the heap limit.

Corrected or Safer Patterns

Bounded caches (LRU, weak or soft references), removing listeners, deterministic resource cleanup, sizing heaps with headroom, measuring retained sizes.

Debugging and Inspection Techniques

GC logs, heap dumps and dominator trees, allocation profilers (tracemalloc, async-profiler, Chrome allocation timelines), gc.get_referrers for leaks in Python.

Performance Considerations

Allocation rate drives young-GC frequency; live-set size drives old-GC cost; barriers cost a little on every reference write; choose collectors by latency versus throughput needs.

Security Considerations

Use-after-free bugs in runtimes (and native extensions) are critical vulnerabilities; resource exhaustion (allocation bombs) needs quotas; heap dumps contain secrets — protect them.

Portability and Version Caveats

Collectors, flags and log formats are implementation-specific (HotSpot's unified logging since JDK 9; CPython's GC thresholds and generations change between versions — 3.13's details differ from 3.12's).

Real-World Use Cases

Tuning JVM services for latency, chasing memory leaks in Node.js servers, reducing allocation in Python data pipelines, choosing Rust or C++ for deterministic memory in real-time systems.

Best Practices

Measure before tuning, prefer short-lived objects, close resources deterministically, bound caches, monitor GC pauses as a service metric.

Exercises

E1 · Beginner. Why does the x ↔ y cycle survive del x, y in CPython with the cycle collector disabled?

Answer: Each object is still referenced by the other, so both reference counts stay at 1; only a tracing pass can see the pair is unreachable from roots.

E2 · Intermediate. Explain why a generational collector needs a write barrier.

Answer: When an old object is updated to reference a young object, a minor collection (which does not scan the old generation) must still treat that young object as reachable; the write barrier records the old object's card/remembered-set entry on every such store.

E3 · Advanced challenge. Design a mark-sweep collector for MiniLang if its VM managed its own heap of closures and upvalues.

Answer: Roots: the value stack, frames' closures, open upvalues, globals and the compiler's in-progress functions. Mark: worklist (gray stack) starting from roots; marking a closure marks its function prototype and upvalues; a prototype marks its constants; a closed upvalue marks its value. Sweep: walk the list of all allocated objects, free unmarked, clear marks. Trigger when bytes allocated exceed a threshold that grows with the live set.

Quiz

? What do tracing collectors start from?
+ Roots: stack slots, globals, registers and handles :: Everything reachable from them is live.
- The oldest object :: Age does not define reachability.
- Objects with a zero reference count :: That is reference counting.
- The program's source code :: No.

? Why is `with open(...)` still needed in a garbage-collected language?
+ GC does not guarantee timely release of OS resources such as file handles. :: Cleanup must be deterministic.
- Files are not objects. :: They are, but the OS handle needs explicit closing.
- GC deletes files. :: No.
- Python has no garbage collector. :: It has reference counting and a cycle collector.

? Which can cause memory leaks in GC'd languages? (select all that apply)
+ An unbounded cache map :: Entries stay reachable.
+ Event listeners never removed :: The emitter keeps them reachable.
- Short-lived temporary objects :: Collected cheaply.
+ A closure capturing a large object stored globally :: Retains the object.

Summary and Key Takeaways

Runtimes reclaim unreachable objects by reference counting (with cycle detection) or tracing (mark-sweep, compacting, copying), usually generationally, with barriers and remembered sets for efficiency and concurrency. GC does not close resources, and reachable garbage still leaks — measure with logs and heap snapshots.