Introduction to GPU Computing & CUDA

CUDA (Compute Unified Device Architecture) is NVIDIA's parallel computing platform and programming model.

1.1 What is CUDA?

CUDA (Compute Unified Device Architecture) is NVIDIA's parallel computing platform and programming model. It lets you write programs that run on NVIDIA GPUs using extensions to C/C++ (plus Fortran, Python, and other languages).

A CUDA program has:

Host code — runs on the CPU (allocates memory, launches work).

Device code — kernels that run on the GPU, executed by thousands of threads in parallel.

1.2 CPU vs GPU

few powerful cores (8–128) thousands of simpler cores optimized for latency (single-thread speed) optimized for throughput (many threads) large caches, branch prediction, out-of-order execution huge register files, hides latency by switching threads good at serial, branchy code good at data-parallel, arithmetic-heavy work ~100 GB/s memory bandwidth 1–8 TB/s (HBM) memory bandwidth

GPUs excel when the same operation is applied to many data elements: linear algebra, deep learning, image/video processing, simulations, cryptography, data analytics.

1.3 History

GeForce 256 — "first GPU" (fixed-function graphics) programmable shaders; GPGPU via graphics APIs

Tesla architecture (G80) — unified shaders CUDA 1.0 released Fermi: caches, ECC, double precision Kepler: dynamic parallelism, Hyper-Q

Maxwell; 2016 Pascal: unified memory page faulting, NVLink, HBM2

Volta: Tensor Cores, independent thread scheduling Turing: RT cores, INT8/INT4 Tensor Cores

Ampere (A100): TF32, BF16, sparsity, MIG, async copy Hopper (H100): TMA, thread block clusters, FP8, Transformer Engine; Ada Lovelace

Blackwell (B200/GB200): FP4/FP6, 5th-gen Tensor Cores, NVLink 5; CUDA 12.x/13

1.4 The CUDA ecosystem

Languages: CUDA C/C++, CUDA Fortran, Python (CuPy, Numba, CUDA Python, Triton), Julia (CUDA.jl).

Libraries: cuBLAS, cuFFT, cuRAND, cuSPARSE, cuSOLVER, cuDNN, NCCL, TensorRT, Thrust, CUB, CUTLASS, RAPIDS.

Tools: nvcc, Nsight Systems, Nsight Compute, cuda-gdb, compute-sanitizer.

Frameworks built on CUDA: PyTorch, TensorFlow, JAX, RAPIDS, cuQuantum, Omniverse.