The Memory Hierarchy

Key performance principle: minimize slow global memory traffic; reuse data from registers and shared memory; access global memory in coalesced patterns.

local scalar variables large/dynamically indexed local arrays, spills cudaMalloc, __device__ L2 (L1 optional) R (from device) R (surface: R/W)

Key performance principle: minimize slow global memory traffic; reuse data from registers and shared memory; access global memory in coalesced patterns.

┌──────────────────────────── GPU ─────────────────────────────┐
│  SM 0                         SM 1               ...         │
│  ┌──────────────────────┐    ┌──────────────────────┐        │
│  │ Registers (per thread)│    │ Registers            │        │
│  │ Shared mem / L1      │    │ Shared mem / L1      │        │
│  └──────────┬───────────┘    └──────────┬───────────┘        │
│             └────────── L2 cache ───────┘                    │
│                         │                                    │
│     Global memory (HBM/GDDR): global, local, constant, texture│
└─────────────────────────┬────────────────────────────────────┘
                          │ PCIe / NVLink
                     Host memory (CPU RAM)