Neural Networks & Backpropagation

How deep learning actually learns. Learn the artificial neuron, activation functions (sigmoid, tanh, ReLU, GELU), multilayer perceptrons and the universal approximation idea, forward pass, loss functions (MSE, cross-entropy), backpropagation via the chain rule, computational graphs and autograd, weight initialisation, vanishing/exploding gradients, batch normalisation, dropout, and training loops in PyTorch — with a NumPy network learning XOR from scratch.

A neural network stacks layers of simple units: each computes a weighted sum of inputs and applies a non-linear activation. With enough units and layers, networks can represent extremely complex functions — images to labels, text to text.

Backpropagation efficiently computes how much every weight contributed to the error, so gradient descent can adjust millions or billions of parameters.

An analogy

Backpropagation is like a company reviewing a failed project: the error is traced back from the final result through each department, and each person's contribution to the mistake is measured so they can adjust — all in one pass backwards through the organisation.

From neuron to network

A neuron computes z = Σwᵢxᵢ + b and outputs φ(z). Without non-linear activations, stacked layers collapse into one linear model — non-linearity lets networks learn curved decision boundaries. Layers: input → hidden layers → output (sigmoid for binary, softmax for classes, linear for regression).

Training loop

Each iteration:

• Take a mini-batch of data
• Forward pass — compute predictions and loss
• Backward pass — compute gradients (backprop / autograd)
• Optimiser step — update weights (SGD, Adam)
• Repeat for many epochs; monitor training and validation loss

Making deep nets trainable

Sigmoid/tanh saturate, causing vanishing gradients in deep nets; ReLU and variants (Leaky ReLU, GELU) fix much of this. Good initialisation (Xavier/Glorot for tanh, He for ReLU) keeps signal variance stable. Batch/layer normalisation stabilise training; residual connections let gradients flow through very deep networks. Gradient clipping prevents explosions.

Regularisation and scale

Dropout randomly zeros units during training; weight decay (L2); data augmentation; early stopping. Modern networks have millions to trillions of parameters, trained on GPUs/TPUs with mixed precision and distributed data/model parallelism. Scaling laws show loss falls predictably with more data, parameters and compute. Frameworks: PyTorch (dynamic graphs), JAX, TensorFlow.

Formulas

• Layer: a⁽ˡ⁾ = φ(W⁽ˡ⁾a⁽ˡ⁻¹⁾ + b⁽ˡ⁾)
• Output error (sigmoid + BCE): δ⁽ᴸ⁾ = ŷ − y
• Backprop: δ⁽ˡ⁾ = (W⁽ˡ⁺¹⁾ᵀδ⁽ˡ⁺¹⁾) ⊙ φ′(z⁽ˡ⁾)
• Weight gradient: ∂L/∂W⁽ˡ⁾ = δ⁽ˡ⁾ a⁽ˡ⁻¹⁾ᵀ
• Activations: ReLU = max(0, z); tanh′ = 1 − tanh²; σ′ = σ(1 − σ)

Worked example: Parameter count

Given: MLP 784 → 128 → 10

Find: Parameters.

• 784 × 128 + 128 + 128 × 10 + 10

Answer: 101 770

Worked example: Neuron output

Given: x = (1, 2), w = (0.5, −0.25), b = 0.1, ReLU

Find: a.

• z = 0.5 − 0.5 + 0.1 = 0.1

Answer: 0.1

Worked example: Sigmoid gradient

Given: σ(z) = 0.9

Find: σ′(z).

• 0.9 × 0.1

Answer: 0.09 (small → vanishing gradients when saturated)

Backpropagation on XOR with NumPy

import numpy as np
# A 2-layer neural network trained with backpropagation on XOR
rng = np.random.default_rng(3)
X = np.array([[0, 0], [0, 1], [1, 0], [1, 1]], float)
y = np.array([[0], [1], [1], [0]], float)
W1 = rng.normal(0, 1, (2, 4)); b1 = np.zeros(4)
W2 = rng.normal(0, 1, (4, 1)); b2 = np.zeros(1)
sig = lambda z: 1 / (1 + np.exp(-z))
for step in range(5001):
    h = np.tanh(X @ W1 + b1)                 # forward
    out = sig(h @ W2 + b2)
    loss = -np.mean(y * np.log(out) + (1 - y) * np.log(1 - out))
    d_out = (out - y) / len(X)               # backward (BCE + sigmoid)
    dW2 = h.T @ d_out; db2 = d_out.sum(0)
    d_h = d_out @ W2.T * (1 - h ** 2)
    dW1 = X.T @ d_h; db1 = d_h.sum(0)
    for p, g in ((W1, dW1), (b1, db1), (W2, dW2), (b2, db2)):
        p -= 1.0 * g
    if step in (0, 1000, 5000):
        print(f"step {step:4d}: loss {loss:.4f}")
print("predictions:", np.round(out.ravel(), 3).tolist())
print(f"parameters: {W1.size + b1.size + W2.size + b2.size}")
# step    0: loss 1.3850
# step 1000: loss 0.0023
# step 5000: loss 0.0004
# predictions: [0.0, 1.0, 0.999, 0.001]
# parameters: 17

XOR can't be separated by a single line, so logistic regression fails — but one hidden layer of 4 tanh units solves it. The backward pass applies the chain rule layer by layer; frameworks like PyTorch compute these gradients automatically (autograd), but the arithmetic is exactly this.

Common mistakes

• Wrong: No non-linearity between layers. — Right: Stacked linear layers are just one linear map.
• Wrong: Initialising all weights to zero. — Right: Symmetry never breaks — use random init.
• Wrong: Ignoring validation loss. — Right: Stop when validation loss rises (overfitting).

Applications

• Tabular & sensor models: MLPs for regression and classification.
• Recommendation: Deep embedding models.
• Control & robotics: Neural policies.
• Everything deep: CNNs, RNNs, transformers build on these ideas.

Key takeaways

• Layers: a = φ(Wx + b); non-linearity is essential.
• Backprop = chain rule, layer by layer.
• ReLU, good init, normalisation, residuals.
• Regularise: dropout, weight decay, early stopping.

Going further

• Automatic differentiation: Reverse-mode AD in frameworks.
• Universal approximation theorem: What width/depth can represent.
• Scaling laws: Loss vs parameters, data and compute.