Advanced Memory Architecture and Research Topics
This module looks at where memory systems are heading and at the techniques that power large servers, operating-system kernels, language runtimes and security hardware.
Learning Objectives
Explain NUMA locality, huge pages, device memory attributes, cache-coherence protocols (MESI family), memory consistency models and TLB shootdowns.
Describe speculative-execution side channels conceptually and their mitigations.
Explain kernel slab allocators, page colouring, persistent memory and its programming concerns, memory compression and deduplication.
Compare garbage-collection algorithms (mark-sweep, mark-compact, copying, generational, concurrent) and concurrent allocation with thread-local caches.
Explain safe memory reclamation for lock-free structures (hazard pointers, epochs, RCU) and distinguish reclamation from destruction.
Survey hardware and language trends: memory tagging, pointer authentication, control-flow integrity, capabilities (CHERI), formal verification and memory-safe systems languages.
Apply rigorous performance methodology.
Plain-Language Explanation
This module looks at where memory systems are heading and at the techniques that power large servers, operating-system kernels, language runtimes and security hardware. Each topic is an active area of engineering and research; the goal is a correct mental model and pointers to go deeper, not every detail. Topics that depend on specific current hardware are marked as such — they change from one CPU generation to the next.
Precise Technical Explanation
NUMA and locality On multi-socket machines each socket's memory controller serves its local DIMMs; remote accesses cross an interconnect (UPI, Infinity Fabric) and cost roughly 1.5–2× the latency with lower bandwidth. Linux allocates on the *first-touch* node; numactl --cpunodebind --membind, mbind, set_mempolicy and NUMA-aware allocators keep threads and their data together. Many large single-socket CPUs (chiplet designs) expose sub-NUMA clusters too.
2 MiB (and 1 GiB) pages cut TLB misses and page-table memory; THP makes them automatic, at the cost of compaction work and possible memory bloat. The experiment below shows the kernel honouring madvise(MADV_HUGEPAGE) on the build machine.
Memory-mapped I/O and device memory attributes Device registers must not be cached, combined or reordered arbitrarily. Arm's memory types (Normal, Device-nGnRnE … Device-GRE) and x86's memory types (UC, WC, WT, WB via PAT/MTRRs) tell the CPU how to treat each region; write-combining speeds up frame buffers. Drivers map device memory with the right attributes (ioremap, ioremap_wc).
Cache coherence protocols (MESI-family overview) Each cache line in each core is Modified (dirty, exclusive), Exclusive (clean, only copy), Shared (clean, possibly in other caches) or Invalid. Reads of an Invalid line fetch it (from memory or another cache); writes require ownership, invalidating other copies. Extensions add Owned (MOESI, AMD: share dirty data without writing back) and Forward (MESIF, Intel: one sharer answers requests). Large systems use directory-based or snoop-filtered protocols instead of broadcasting. A simulator below traces the states.
Memory consistency models The *consistency model* defines which orders of loads and stores other cores may observe: sequential consistency (simple, slow), TSO (x86: stores may be delayed behind later loads), weak/relaxed models (Arm, RISC-V RVWMO, POWER) with explicit barriers. Language models (C/C++11, Java) are defined in terms of happens-before and map onto each hardware model (Module 14).
Speculative execution and side-channel concepts Out-of-order CPUs execute instructions that are later squashed, but their effects on caches and other micro-architectural state can be measured through timing. Spectre-class issues make the victim speculatively access secret data; Meltdown-class issues let speculation bypass permission checks. Mitigations: kernel page-table isolation, retpolines and IBRS/eIBRS, speculation barriers (lfence, csdb), index masking, site isolation in browsers, and hardware fixes in newer CPUs. This course discusses the concepts only.
When the kernel changes or removes a mapping that other cores may have cached in their TLBs, it must send inter-processor interrupts so they invalidate the entry — a TLB shootdown. Frequent munmap/mprotect in many-threaded programs makes them expensive; batching, per-thread arenas and Arm's broadcast TLBI instructions reduce the cost.
Kernel allocators and slab concepts Kernels allocate pages with a buddy allocator (power-of-two blocks, cheap splitting and merging) and objects with slab allocators (SLAB/SLUB in Linux): caches of same-sized objects carved from pages, with per-CPU free lists. /proc/slabinfo and slabtop show them.
Page coloring (historical/architecture-dependent) In physically indexed caches, the physical page frame decides which cache sets a page uses ("colour"). OSes once chose frames to spread colours evenly and reduce conflict misses; modern systems mostly rely on high associativity, although colouring is used for cache partitioning in real-time and security contexts (and Intel CAT / Arm MPAM provide hardware partitioning).
Persistent memory concepts Byte-addressable non-volatile memory (Intel Optane DC PMEM — now discontinued, CXL-attached persistent memory, battery-backed DRAM) mapped into the address space with DAX: data survives power loss without block I/O. The CXL standard is the current vehicle for memory expansion and pooling.
Non-volatile memory programming concerns Stores sit in volatile CPU caches until written back; programs must flush (clwb/clflushopt) and fence (sfence) to make data durable, and design updates so a crash at any point leaves consistent data (logging, copy-on-write, 8-byte atomic persistence). Libraries such as PMDK encapsulate this.
Memory compression and deduplication zswap/zram (Linux) and memory compression (Windows, macOS) compress cold pages in RAM instead of swapping them to disk. Kernel same-page merging (KSM) and hypervisor page sharing deduplicate identical pages between VMs or processes (with copy-on-write) — saving memory, at some CPU and side-channel cost.
Garbage collector algorithms mark reachable objects, sweep the rest into free lists simple, no moving fragmentation, sweep cost ∝ heap mark, then slide live objects together extra passes, pointer updates Copying (semi-space) copy live objects to an empty space fast allocation, cost ∝ live data half the space idle young objects in a copying nursery, old ones collected rarely exploits "most objects die young" write barriers, remembered sets Concurrent / incremental do marking (and compaction) while the program runs barriers, more CPU, floating garbage
A runnable mark-sweep collector is shown below.
Concurrent allocation and thread-local allocation caches Allocators and GCs give each thread a private buffer (TLAB in the JVM, allocation contexts in.NET, tcache in glibc, thread caches in jemalloc/tcmalloc/mimalloc) so most allocations need no synchronisation — a pointer bump or a pop from a local list.
Lock-free data structures and reclamation: hazard pointers, epochs, RCU
In a lock-free structure, a thread may still be reading a node after another thread unlinked it — freeing it immediately would be a use-after-free (and a reused address enables the ABA problem). Safe-reclamation schemes defer the free:
Hazard pointers — each reader publishes the pointers it is using; a node is freed only when no hazard pointer refers to it (standardised for C++26).
Epoch-based reclamation — readers announce the global epoch they entered; retired nodes are freed once every thread has moved past the epoch in which they were retired.
RCU (read-copy-update) — readers run without locks inside read-side critical sections; writers publish a new version and free the old one after a *grace period* in which all pre-existing readers have finished. Used throughout the Linux kernel.
Memory reclamation versus object destruction *Destruction* ends an object's logical lifetime (destructor, Dispose, unlinking from a structure); *reclamation* returns its memory for reuse. In lock-free code and GC languages they happen at different times — an object can be logically dead but not yet safe to reclaim (or, with finalizers, reclaimed long after it was last used).
Memory tagging and pointer authentication concepts Memory tagging (Arm MTE, SPARC ADI): every 16-byte granule has a 4-bit tag, and pointers carry a tag in unused top bits; a mismatch on access traps (synchronously or asynchronously). It detects use-after-free and overflows probabilistically in production at low cost. Android supports MTE on capable devices.
Pointer authentication (Arm PAC): a cryptographic signature in the pointer's top bits, checked before use, protects return addresses and function pointers against corruption.
Control-flow integrity and memory-safe architecture trends Control-flow integrity restricts indirect branches to valid targets: compiler CFI (Clang -fsanitize=cfi, Microsoft CFG/XFG), hardware landing pads (Intel IBT endbr64 — seen in every listing in this course — and Arm BTI) and shadow stacks (Intel CET, Arm GCS) for returns.
Capability-based memory protection overview CHERI (Arm Morello prototype, CHERI-RISC-V, CHERIoT for microcontrollers) replaces plain pointers with hardware *capabilities*: 128-bit values carrying bounds, permissions and a validity tag that software cannot forge. Out-of-bounds accesses and pointer forgery trap deterministically; C/C++ code mostly recompiles unchanged.
Formal verification and memory safety Proving the absence of memory errors mathematically: seL4 (a verified microkernel), CompCert (a verified C compiler), Frama-C and SPARK (proofs about C and Ada programs), Rust verifiers (Kani, Prusti, Verus), and model checking of concurrent algorithms (TLA+, GenMC).
Safe systems programming and language design Rust (ownership), Swift (ARC + exclusivity), Ada/SPARK, and efforts to make C++ safer (hardened standard libraries, lifetime profiles, bounds-checked modes), plus government and industry guidance encouraging memory-safe languages for new code.
Performance trade-offs and measurement methodology State a hypothesis, build a model ("this should be bandwidth-bound at ~20 GB/s"), measure with controlled experiments (repeated runs, fixed frequency, isolated cores, realistic data), report distributions and confidence, and confirm the mechanism with counters. Beware coordinated omission in latency measurements and benchmarking only the happy path.
Current hardware/OS-specific features as optional updateable lessons
Features in flux — CXL memory tiering, Arm MTE deployment, Intel/AMD memory encryption (TME, SME/SEV, TDX), Linux multi-generational LRU and memory folios, 5-level paging, new GC algorithms (generational ZGC) — change quickly. Treat statements about them as dated, check the vendor's current documentation, and expect this section to be revised over time.
Why It Matters in Real Systems
Cloud providers, database engines, JVM/.NET/V8 teams, kernel developers and security engineers work directly with these mechanisms. Even application developers meet them as NUMA slowdowns, THP latency spikes, GC pauses, side-channel advisories and MTE-detected bugs.
Key Terms
Memory access time depends on which node owns the memory MESI / MOESI / MESIF Cache-coherence state protocols
Rules for observable memory-operation orders Cross-core TLB invalidation after a mapping change Kernel cache of same-sized objects
Byte-addressable memory that survives power loss Mark-sweep / copying / generational GC Hazard pointer / epoch / RCU Safe memory-reclamation schemes
Arm memory tagging / pointer authentication Capability hardware with bounded, unforgeable pointers
Memory Layout and Execution Flow
flowchart LR
I[Invalid] -- read, no other copy --> E[Exclusive]
I -- read, others have it --> S[Shared]
I -- write --> M[Modified]
E -- write --> M
S -- write: invalidate others --> M
E -- other core reads --> S
M -- other core reads: write back --> S
S -- other core writes --> I
M -- other core writes --> ISyntax
madvise(p, len, MADV_HUGEPAGE); /* Linux: request THP for a region */
numa_alloc_onnode(size, node); /* libnuma: allocate on a NUMA node */
_mm_clwb(addr); _mm_sfence(); /* x86: write back a line to persistence */std::hazard_pointer hp = std::make_hazard_pointer(); // C++26 hazard pointersComplete Runnable Example
Huge pages in practice — the same 64 MiB touched with and without MADV_HUGEPAGE (Linux, THP in madvise mode on the build machine):
Without advice each 4 KiB page faults separately (16,384 faults). With MADV_HUGEPAGE the kernel backed the region with 2 MiB pages: about 32 faults and 64 MiB of AnonHugePages — and the TLB needs 32 entries instead of 16,384 to cover it.
A mark-sweep garbage collector — a toy heap in Python that shows roots, marking, sweeping and why cycles are not a problem for tracing collectors:
MESI states — a small simulator tracing one cache line through reads and writes by three cores:
#include <stdint.h>
#include <stdio.h>
#include <string.h>
#include <sys/mman.h>
#include <sys/resource.h>
#define SIZE (64u << 20)
#define HUGE (2u << 20)
static long faults(void) { struct rusage r; getrusage(RUSAGE_SELF, &r); return r.ru_minflt; }
static long anon_huge_kib(void) /* AnonHugePages of this process */
{
FILE *f = fopen("/proc/self/smaps_rollup", "r");
char line[256];
long kib = 0;
while (f != NULL && fgets(line, sizeof line, f) != NULL) {
if (strncmp(line, "AnonHugePages:", 14) == 0) sscanf(line + 14, "%ld", &kib);
}
if (f != NULL) fclose(f);
return kib;
}
static void run(int advise)
{
/* over-allocate so we can align the region to a 2 MiB boundary */
unsigned char *raw = mmap(NULL, SIZE + HUGE, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
if (raw == MAP_FAILED) return;
unsigned char *p = (unsigned char *)(((uintptr_t)raw + HUGE - 1) & ~(uintptr_t)(HUGE - 1));
if (advise) madvise(p, SIZE, MADV_HUGEPAGE);
long f0 = faults();
for (size_t i = 0; i < SIZE; i += 4096) p[i] = 1;
long f1 = faults();
printf("%-22s minor faults %6ld, AnonHugePages %6ld KiB\n",
…without advice: minor faults 16384, AnonHugePages 0 KiB
with MADV_HUGEPAGE: minor faults 32, AnonHugePages 65536 KiBclass Obj:
def __init__(self, name):
self.name, self.refs, self.marked = name, [], False
class Heap:
def __init__(self):
self.objects, self.roots = [], []
def new(self, name):
o = Obj(name)
self.objects.append(o)
return o
def mark(self):
stack = list(self.roots) # explicit stack: no recursion limit
while stack:
o = stack.pop()
if not o.marked:
o.marked = True
stack.extend(o.refs)
def sweep(self):
freed = [o.name for o in self.objects if not o.marked]
self.objects = [o for o in self.objects if o.marked]
for o in self.objects:
o.marked = False # reset for the next cycle
return freed
def collect(self):
self.mark()
return self.sweep()
h = Heap()
config, cache, session = h.new("config"), h.new("cache"), h.new("session")
a, b = h.new("cycleA"), h.new("cycleB")
temp = h.new("temp")
config.refs.append(cache)
session.refs.append(temp)
a.refs.append(b); b.refs.append(a) # an unreachable cycle
h.roots = [config, session] # e.g. a glob
…heap: ['config', 'cache', 'session', 'cycleA', 'cycleB', 'temp']
collect 1 freed: ['cycleA', 'cycleB']
collect 2 freed: ['temp']
collect 3 freed: ['session']
live: ['config', 'cache']def access(states, core, op):
"""Apply one read (R) or write (W) by `core` to a single cache line; return the bus transaction."""
others = [c for c in range(len(states)) if c != core]
mine = states[core]
if op == "R":
if mine != "I":
return "none (hit)"
if any(states[c] != "I" for c in others):
for c in others:
if states[c] in "ME":
states[c] = "S" # owner supplies the data (M also writes back) and downgrades
states[core] = "S"
return "BusRd (data from another cache)"
states[core] = "E" # nobody else has it: exclusive
return "BusRd (data from memory)"
if mine == "M":
return "none (hit, already owned)"
if mine == "E":
states[core] = "M"
return "none (silent E->M upgrade)"
for c in others:
states[c] = "I" # invalidate every other copy
states[core] = "M"
return "BusUpgr (invalidate others)" if mine == "S" else "BusRdX (read for ownership)"
states = ["I", "I", "I"]
trace = [(0, "R"), (0, "W"), (1, "R"), (1, "W"), (2, "R"), (0, "W"), (0, "W")
…op core0 core1 core2 bus transaction
P0 R E I I BusRd (data from memory)
P0 W M I I none (silent E->M upgrade)
P1 R S S I BusRd (data from another cache)
P1 W I M I BusUpgr (invalidate others)
P2 R I S S BusRd (data from another cache)
P0 W M I I BusRdX (read for ownership)
P0 W M I I none (hit, already owned)
P2 R S I S BusRd (data from another cache)Line-by-Line Explanation
Huge pages: the region is aligned to 2 MiB (THP can only use naturally aligned 2 MiB ranges), madvise marks it eligible, and the fault counter and AnonHugePages show which page size the kernel used.
Mark-sweep: marking walks from the roots with an explicit stack; sweeping frees everything unmarked. The cycleA/cycleB cycle is freed in the first collection because nothing reachable points to it — unlike reference counting (Module 6). temp dies once session drops it, and session dies once it is no longer a root.
MESI: the first read finds no other copy (E), so the following write upgrades silently to M; a second core's read makes the owner supply the data and both become S; a write to an S line must invalidate the other copy (BusUpgr); a write to an I line needs a read-for-ownership (BusRdX); repeated writes by the owner need no bus traffic at all.
Expected Output
The outputs are shown under each program above.
Step-by-Step Execution Trace
The MESI table above is itself the trace: each row is one memory operation, the three state columns are the line's state in each core's cache *after* the operation, and the last column is the coherence traffic it caused. Note the row where core 0 writes a line it already holds in M: "none (hit, already owned)" — the reason a core that owns a line can update it at full speed, and why ping-ponging ownership (false sharing) is so expensive.
Interactive Visualization
The cache simulator (Module 13) and the address translator (Module 12) let you explore associativity, replacement and page size — the building blocks of the topics here.
Common Mistakes and Failure Modes
Enabling THP always for latency-sensitive databases without testing (compaction stalls).
Ignoring NUMA placement on multi-socket servers (threads on one node, memory on another).
Writing lock-free code that frees nodes immediately (use-after-free, ABA).
Assuming persistent memory is durable after a plain store.
Treating a mitigation (MTE, CFI, ASLR) as a fix for the bug it detects.
Quoting micro-architectural numbers from one CPU as universal.
Corrected / Safe Pattern
Pin threads and memory together on NUMA systems; first-touch data from the thread that will use it.
Use madvise(MADV_HUGEPAGE) selectively for large, long-lived regions and measure.
Use proven reclamation libraries (folly hazard pointers, crossbeam-epoch, liburcu, the kernel's RCU) rather than hand-rolled schemes.
Use PMDK-style transactional APIs for persistent memory.
Enable MTE/PAC/CFI/shadow stacks where available as defence in depth, and keep fixing bugs they report.
Debugging and Diagnostic Techniques
numastat, numactl --hardware, /proc/<pid>/numa_maps; /proc/meminfo (AnonHugePages), /sys/kernel/mm/transparent_hugepage/*, perf stat -e dTLB-load-misses; perf c2c for coherence traffic; slabtop; JVM -Xlog:gc* and .NET GC events for collector behavior; MTE reports in Android tombstones; liburcu and kernel lockdep/KCSAN for concurrency.
Performance Implications
NUMA-aware placement and huge pages often give double-digit percentage gains for memory-bound servers; coherence traffic and TLB shootdowns limit scalability; GC choice trades throughput, pause time and memory footprint; safe reclamation adds per-access overhead (hazard pointers) or delayed memory release (epochs, RCU). Every one of these must be measured on the target hardware.
Security and Safety Implications
Side channels, Rowhammer (bit flips induced by repeated DRAM row activation), and memory deduplication leaks show that the memory system itself is attack surface. Hardware features (MTE, PAC, CET, CHERI, memory encryption) and memory-safe languages are the long-term answer to memory-corruption bugs; formal verification gives the strongest guarantees for critical components.
Platform and Language Caveats
MTE, PAC and BTI exist only on recent Arm cores with OS support; CET needs recent Intel/AMD CPUs, kernel and toolchain support; CHERI is available on research/prototype hardware and some microcontrollers.
THP behavior and defaults differ across distributions and kernel versions.
GC algorithms and tuning flags change between JVM/.NET releases.
Real-World Application
Databases pin memory per NUMA node; JVMs use generational ZGC or G1 with region-based copying; the Linux kernel uses SLUB, RCU and per-CPU data; Android ships MTE-enabled builds for testing and production on supported devices; Microsoft and Google invest in CHERI, Rust and memory tagging to remove whole classes of vulnerabilities.
Best Practices Checklist
[ ] Know your platform's NUMA layout, page sizes and THP settings.
[ ] Measure before and after enabling any memory-system feature.
[ ] Use established libraries for lock-free reclamation and persistent memory.
[ ] Keep hardware security features enabled; treat their reports as bugs.
[ ] Revisit hardware-specific advice regularly; it ages fast.
Exercises
E1 · Beginner. Why does a copying collector's cost depend on the amount of *live* data rather than the heap size?
Answer: It only visits and copies reachable objects; dead objects are never touched — the whole from-space is discarded at once.
E2 · Intermediate. In the MESI trace, explain why core 0's second consecutive write produced no bus transaction while the write just before it did.
Answer: Before that first write, core 0's copy was Invalid and core 2 held the line Shared, so core 0 had to obtain the line and invalidate core 2's copy (BusRdX). Afterwards core 0 holds the line in M — the only valid copy — so further writes hit locally.
E3 · Advanced challenge. A lock-free stack pops node N; thread T1 read head = N and N->next = M just before being descheduled; meanwhile N is popped, freed, reallocated and pushed again. Explain how T1's CAS can wrongly succeed and how hazard pointers prevent it.
Answer: This is ABA: when T1 resumes, head equals N again (same address), so CAS(head, N, M) succeeds and installs the stale M, corrupting the stack (M may already be freed). With hazard pointers, T1 publishes N as hazardous before reading N->next; N cannot be freed — and therefore not reused at the same address — while the hazard pointer exists, so the ABA sequence cannot occur.
Quiz
? Why did MADV_HUGEPAGE reduce the fault count from 16,384 to about 32?
+ Each fault mapped a 2 MiB huge page instead of a 4 KiB page. :: 64 MiB / 2 MiB = 32.
- The kernel skipped zeroing the memory. :: Huge pages are zeroed too.
- madvise pre-faulted the whole region. :: Faults still happened, just fewer.
- The memory was swapped out. :: Nothing was swapped.
? Which garbage-collection approaches reclaim unreachable cycles? (select all that apply)
+ Mark-sweep :: Unreachable objects are never marked.
+ Copying :: Only reachable objects are copied.
- Pure reference counting :: Cycles keep their counts above zero.
+ Generational collectors built on tracing :: They trace reachability per generation.
? What problem do hazard pointers, epochs and RCU solve?
- Deadlocks between mutexes :: They are about memory reclamation, not locking.
+ Freeing nodes of lock-free structures while other threads may still read them :: They defer reclamation safely.
- False sharing :: That is a layout problem.
- Garbage collection pauses :: They are manual reclamation schemes.
? In MESI, what happens when a core writes a line that other cores hold in Shared?
- The write proceeds and the other copies sta
…Summary
Advanced memory systems add NUMA, huge pages, device memory types, coherence protocols, consistency models and TLB shootdowns; kernels use buddy and slab allocators; runtimes choose among mark-sweep, compacting, copying, generational and concurrent collectors with thread-local allocation; lock-free code needs safe reclamation. Security is moving into hardware — tagging, authentication, CFI, capabilities — and into memory-safe languages and verification. Hardware-specific details change quickly: measure and re-check.
References and Further Reading
Daniel Sorin, Mark Hill & David Wood, *A Primer on Memory Consistency and Cache Coherence*, 2nd ed.
Richard Jones, Antony Hosking & Eliot Moss, *The Garbage Collection Handbook*, 2nd ed.
Paul McKenney, "What is RCU, Fundamentally?" (LWN); Maged Michael, "Hazard Pointers" (IEEE TPDS 2004).
Arm, "Memory Tagging Extension" white paper; the CHERI project (University of Cambridge).
Paul Kocher et al., "Spectre Attacks: Exploiting Speculative Execution" (2019).