Transformers & Large Language Models

The architecture behind modern AI. Learn tokenisation (BPE), embeddings and positional encodings, scaled dot-product attention (queries, keys, values), multi-head attention, causal masking, the transformer block (attention + MLP + residual + layer norm), encoder vs decoder models, pre-training by next-token prediction, scaling, instruction tuning and RLHF, decoding (greedy, temperature, top-p), context windows and KV caching, and the capabilities and limitations of LLMs (hallucination) — with NumPy code for causal self-attention and softmax temperature.

Transformers process sequences with attention: every token looks at every other token and decides what is relevant. This parallelises well on GPUs and captures long-range context — the foundation of GPT, Claude, Gemini, Llama and modern vision and speech models.

Large language models are transformers trained on enormous text corpora to predict the next token, then refined with instruction tuning and human feedback to become helpful assistants.

An analogy

Attention is like a group discussion where each person (token) asks a question (query), everyone holds up a name card describing what they know (key), and the asker listens mostly to the most relevant people, blending what they say (values) into their own understanding.

Tokens and embeddings

Text is split into tokens (sub-words via byte-pair encoding: 'unbelievable' → 'un', 'believ', 'able'); roughly 0.75 English words per token. Each token ID maps to a learned embedding vector (e.g. 4096 dimensions). Positional information is added (sinusoidal, learned or rotary embeddings/RoPE) because attention itself ignores order.

The transformer block

Repeated N times (e.g. 32–120 layers):

Training LLMs

Pre-training: next-token prediction on trillions of tokens — the model learns grammar, facts, reasoning patterns. Supervised fine-tuning (instruction tuning) on prompt–response pairs; preference optimisation (RLHF, DPO) aligns outputs with human preferences for helpfulness and harmlessness. Scaling laws guide the balance of parameters and data (compute-optimal training).

Inference and limitations

Generation is autoregressive: predict, sample, append, repeat. Decoding: greedy, temperature, top-k, top-p (nucleus). KV caching stores past keys/values to avoid recomputation; attention cost grows with context length (O(n²)), addressed by FlashAttention, sparse/linear attention and long-context techniques. Quantisation (INT8/INT4) and speculative decoding speed inference. Limitations: hallucinations, knowledge cut-off, sensitivity to prompts, bias, prompt injection — mitigated with retrieval (RAG), tools, evaluation and guardrails.

Formulas

Worked example: Token estimate

Given: 3000-word document

Find: Tokens.

Answer: ≈ 4000 tokens

Worked example: Parameters

Given: d_model 4096, 32 layers, vocab 32 000

Find: Approx. parameters.

Answer: ≈ 6.6 B (a '7B' model)

Worked example: Attention cost

Given: Context 8k → 32k tokens

Find: Attention compute ratio.

Answer: 16× more (O(n²))

Causal self-attention and temperature

import numpy as np
# Scaled dot-product self-attention on 4 toy token embeddings, plus softmax temperature
rng = np.random.default_rng(0)
tokens = ["the", "cat", "sat", "down"]
d = 8
X = rng.normal(0, 1, (4, d))
Wq, Wk, Wv = (rng.normal(0, 1 / np.sqrt(d), (d, d)) for _ in range(3))
Q, K, V = X @ Wq, X @ Wk, X @ Wv
scores = Q @ K.T / np.sqrt(d)
mask = np.triu(np.ones((4, 4)), 1) * -1e9       # causal mask: no looking ahead
weights = np.exp(scores + mask); weights /= weights.sum(1, keepdims=True)
print("causal attention weights:")
for t, row in zip(tokens, weights):
    print(f"{t:5s}", np.round(row, 2).tolist())
out = weights @ V
print(f"output shape {out.shape}")
logits = np.array([2.0, 1.0, 0.5, -1.0])
for T in (0.5, 1.0, 2.0):
    p = np.exp(logits / T); p /= p.sum()
    print(f"temperature {T}: next-token probs {np.round(p, 3).tolist()}")
# causal attention weights:
# the   [1.0, 0.0, 0.0, 0.0]
# cat   [1.0, 0.0, 0.0, 0.0]
# sat   [0.31, 0.34, 0.35, 0.0]
# down  [0.22, 0.28, 0.21, 0.29]
# output shape (4, 8)
# temperature 0.5: next-token probs [0.842, 0.114, 0.042, 0.002]
# temperature 1.0: next-token probs [0.609, 0.224, 0.136, 0.03]
# temperature 2.0: next-token probs [0.434, 0.263, 0.205, 0.097]

Each row shows where a token looks: the causal mask forces zeros above the diagonal, so 'the' can only see itself and later tokens mix information from all earlier ones (with random, untrained weights here 'cat' happens to put almost all its weight on 'the' — values are rounded to 2 decimals). Temperature reshapes the next-token distribution: low T is focused and deterministic, high T more diverse and creative.

Common mistakes

Applications

Key takeaways

Going further