Linear & Logistic Regression with Gradient Descent

The foundation of machine learning. Learn linear regression (model, mean squared error, closed-form least squares), gradient descent (gradients, learning rate, convergence, batch/mini-batch/stochastic), feature scaling, polynomial features, regularisation (L2 ridge, L1 lasso), logistic regression for classification (sigmoid, log-loss/cross-entropy, decision boundary, probability calibration), softmax for multiclass — with runnable NumPy code implementing both models from scratch and checking against least squares.

Linear models are simple, fast and interpretable — and they introduce every core idea of ML: a parametric model, a loss function, and an optimiser that adjusts parameters to reduce loss.

Neural networks are essentially stacks of linear models with non-linear activations, trained with exactly the same gradient-descent idea.

An analogy

Gradient descent is like walking downhill in fog: you can't see the valley, but you can feel the slope under your feet and take a step downhill. The learning rate is your step size — too small and you crawl, too large and you overshoot and bounce around.

Fitting a line

Linear regression finds the line (or hyperplane) that minimises the average squared distance to the data points. Squared error penalises big mistakes heavily and gives a smooth bowl-shaped loss with a single minimum, which can be solved exactly (normal equation) or iteratively (gradient descent).

Gradient descent in practice

Key choices:

• Learning rate — too high diverges/oscillates, too low is slow; use schedules
• Batch GD (all data), stochastic GD (one sample), mini-batch (e.g. 32–1024) — the standard
• Feature scaling (standardise to mean 0, std 1) makes the loss bowl round and GD fast
• Convergence checks — loss plateau, gradient norm, early stopping on validation
• Momentum and Adam accelerate convergence (see deep learning lesson)

Logistic regression

For classification, pass the linear score through a sigmoid to get a probability; train with log-loss (cross-entropy), whose gradient is simply (p − y)·x. The decision boundary wx + b = 0 is linear; non-linear boundaries need feature engineering (polynomials, interactions) or non-linear models. Softmax generalises to many classes.

Regularisation and interpretation

With many features, models overfit; ridge (L2) shrinks weights smoothly, lasso (L1) drives some to exactly zero (feature selection), elastic net combines both. Coefficients are interpretable (effect per unit of feature, or log-odds for logistic) — valued in finance, medicine and policy. Watch for multicollinearity, outliers (use Huber loss), and non-linearity (check residual plots). Generalised linear models extend to counts (Poisson) and more.

Formulas

• Linear model: ŷ = w·x + b (vector form ŷ = Xw)
• MSE loss: L = (1/N) Σ (ŷᵢ − yᵢ)²
• Gradient step: w ← w − η·∂L/∂w, ∂L/∂w = (2/N) Σ (ŷᵢ − yᵢ)xᵢ
• Normal equation: w = (XᵀX)⁻¹Xᵀy
• Logistic model & log-loss: p = σ(wx + b); L = −(1/N) Σ [y log p + (1 − y) log(1 − p)]
• Ridge: L + λ‖w‖²

Worked example: One gradient step

Given: x = 2, y = 7, w = 1, b = 0, η = 0.1 (single sample, MSE)

Find: New w.

• ŷ = 2, error = −5
• ∂L/∂w = 2 × (−5) × 2 = −20
• w = 1 + 2

Answer: w = 3

Worked example: Sigmoid

Given: z = 0.42 + 3.5 × 0.5

Find: p.

• z = 2.17
• 1/(1 + e^(−2.17))

Answer: 0.897

Worked example: Log-loss

Given: y = 1, p = 0.9

Find: Loss.

• −ln 0.9

Answer: 0.105 (p = 0.1 would give 2.30)

Linear and logistic regression from scratch

import numpy as np
# Linear regression by gradient descent, then logistic regression for classification
rng = np.random.default_rng(0)
X = rng.uniform(0, 10, 100)
y = 3.0 * X + 5.0 + rng.normal(0, 2.0, 100)
w, b, lr = 0.0, 0.0, 0.01
for epoch in range(2000):
    pred = w * X + b
    err = pred - y
    w -= lr * 2 * np.mean(err * X)
    b -= lr * 2 * np.mean(err)
mse = np.mean((w * X + b - y) ** 2)
print(f"gradient descent: w = {w:.3f}, b = {b:.3f}, MSE = {mse:.2f}")
A = np.c_[X, np.ones_like(X)]
print("closed form (least squares):", np.round(np.linalg.lstsq(A, y, rcond=None)[0], 3))
# Logistic regression on a 1-D problem
Xc = rng.normal(0, 1, 200); yc = (Xc + rng.normal(0, 0.5, 200) > 0).astype(float)
wc, bc = 0.0, 0.0
for _ in range(3000):
    p = 1 / (1 + np.exp(-(wc * Xc + bc)))
    wc -= 0.1 * np.mean((p - yc) * Xc); bc -= 0.1 * np.mean(p - yc)
acc = np.mean(((1 / (1 + np.exp(-(wc * Xc + bc)))) > 0.5) == yc)
print(f"logistic: w = {wc:.2f}, b = {bc:.2f}, training accuracy {acc:.3f}")
# gradient descent: w = 2.974, b = 4.985, MSE = 3.76
# closed form (least squares): [2.974 4.985]
# logistic: w = 3.50, b = 0.42, training accuracy 0.865

Gradient descent recovers the true slope (3) and intercept (5) within noise and matches the exact least-squares solution. MSE ≈ 3.76 is close to the noise variance (2² = 4) — the irreducible error. The logistic model learns a steep sigmoid; 86.5% accuracy reflects the label noise built into the data.

Common mistakes

• Wrong: Not scaling features for gradient descent. — Right: Standardise — converges much faster.
• Wrong: Using MSE for classification. — Right: Use log-loss with sigmoid/softmax.
• Wrong: Interpreting coefficients with correlated features. — Right: Check multicollinearity; use regularisation.

Applications

• Demand forecasting: Linear models with engineered features.
• Credit scoring: Interpretable logistic scorecards.
• A/B testing: Regression to estimate effects.
• Baselines: First model in almost every project.

Key takeaways

• Model + loss + optimiser = ML.
• w ← w − η∇L; tune η, scale features.
• Logistic: sigmoid + log-loss for classes.
• Regularise to generalise.

Going further

• Convexity: Why linear/logistic losses have a global minimum.
• Bayesian linear regression: Uncertainty over weights.
• Generalised linear models: Poisson, gamma and link functions.