GPU Architecture

A GPU contains many SMs (e.g., 108 on A100, 132 on H100 SXM).

2.1 Streaming Multiprocessors (SMs)

A GPU contains many SMs (e.g., 108 on A100, 132 on H100 SXM). Each SM has:

CUDA cores (FP32/INT32 units), FP64 units, Tensor Cores, special function units (SFUs: sin, exp, rsqrt).

Warp schedulers (typically 4 per SM).

A large register file (64K 32-bit registers per SM).

L1 cache / shared memory (configurable, up to 228 KB on H100).

Load/store units, texture units.

All SMs share the L2 cache and global memory (GDDR or HBM).

2.2 SIMT execution & warps

Threads are grouped into warps of 32 threads.

A warp executes one instruction at a time for all its threads: SIMT (Single Instruction, Multiple Threads).

If threads in a warp take different branches (divergence), paths are executed serially with inactive threads masked.

The SM hides memory latency by switching between many resident warps each cycle — so GPUs need lots of threads (high occupancy or enough instruction-level parallelism).

Since Volta, threads have independent program counters (independent thread scheduling) — warp-synchronous assumptions require explicit sync primitives.

2.3 Mapping software to hardware

executed by a CUDA core lane warp (32 threads) scheduled by a warp scheduler runs entirely on one SM (never split) spread across all SMs cluster (Hopper+) group of blocks on a GPU Processing Cluster (GPC)

Blocks are scheduled onto SMs as resources (registers, shared memory, thread slots) permit; a block stays on its SM until it finishes.

2.4 Memory hierarchy (overview)

Size (typical) 255 per thread max ~registers latency (~20–30 cycles) up to 228 KB/SM combined with shared

40–50 MB (A100/H100), larger on Blackwell Global memory (HBM/GDDR) device + host-accessible slow (~400–800 cycles) device, read-only, cached fast when broadcast thread (spills) as slow as global