Introduction to AI Engineering & the ML Workflow

From idea to deployed intelligence. Learn what AI, machine learning, deep learning and generative AI are and how they relate, the types of learning (supervised, unsupervised, self-supervised, reinforcement), the end-to-end ML workflow (problem framing, data collection and labelling, splitting, baselines, training, evaluation, deployment, monitoring), features and labels, overfitting and generalisation, the AI engineer's toolkit (Python, NumPy, pandas, scikit-learn, PyTorch, cloud, vector DBs) and responsible AI — with a runnable example that splits data, builds a baseline and learns a simple rule.

Artificial intelligence builds systems that perform tasks needing 'intelligence' — recognising images, understanding language, making predictions and decisions. Machine learning does this by learning patterns from data rather than following hand-written rules.

AI engineering is the discipline of turning models into reliable products: framing problems, preparing data, choosing and training models, evaluating them honestly, deploying them at scale and monitoring them over time.

An analogy

Traditional programming is like giving a cook a recipe. Machine learning is like showing a cook hundreds of finished dishes with their ingredients and letting them figure out the recipe — then testing whether they can cook new dishes they've never seen.

AI, ML, DL and GenAI

Nested fields:

Types of learning

Supervised: learn x → y from labelled examples (classification, regression). Unsupervised: find structure without labels (clustering, dimensionality reduction, anomaly detection). Self-supervised: create labels from the data itself (predict the next word) — the basis of foundation models. Reinforcement learning: learn actions from rewards (games, robotics, RLHF for LLMs).

The ML workflow

1) Frame the problem and success metric; 2) collect, clean and label data; 3) split into train/validation/test (no leakage!); 4) build a simple baseline; 5) engineer features or choose architectures; 6) train and tune hyperparameters on validation; 7) evaluate once on test; 8) deploy; 9) monitor drift and performance; 10) retrain. Most time is spent on data, not models.

Generalisation and responsibility

Models must generalise beyond training data: overfitting (memorising noise) vs underfitting (too simple). Regularisation, more data, cross-validation and early stopping help. Responsible AI covers fairness, privacy, transparency, safety, security (adversarial and prompt-injection attacks) and compliance (EU AI Act, India's DPDP Act). An AI engineer's stack: Python, NumPy/pandas, scikit-learn, PyTorch/JAX, Hugging Face, vector databases, Docker/Kubernetes, cloud ML platforms, experiment tracking.

Formulas

Worked example: Baseline accuracy

Given: 95% of emails are not spam

Find: Accuracy of 'never spam'.

Answer: 95% — so accuracy alone is misleading

Worked example: Data split

Given: 10 000 samples, 80/10/10

Find: Sizes.

Answer: 8000 train, 1000 validation, 1000 test

Worked example: Generalisation gap

Given: Train error 2%, test 15%

Find: Diagnosis.

Answer: Overfitting — regularise or get more data

Split, baseline and a learned rule

import random
# The ML workflow in miniature: data -> split -> baseline -> simple model -> evaluate
random.seed(42)
data = []
for _ in range(200):
    hours = random.uniform(0, 10)                       # study hours
    passed = 1 if hours + random.gauss(0, 1.5) > 5 else 0
    data.append((hours, passed))
random.shuffle(data)
train, test = data[:150], data[150:]
majority = round(sum(y for _, y in train) / len(train))
base_acc = sum(1 for _, y in test if y == majority) / len(test)
# "Learn" the best threshold on training data
best_t = max((t / 10 for t in range(0, 101)), key=lambda t: sum((x > t) == y for x, y in train))
acc = sum((x > best_t) == y for x, y in test) / len(test)
print(f"train {len(train)}, test {len(test)}")
print(f"baseline (always predict {majority}) accuracy {base_acc:.2f}")
print(f"learned rule: pass if hours > {best_t:.1f} -> test accuracy {acc:.2f}")
# train 150, test 50
# baseline (always predict 1) accuracy 0.60
# learned rule: pass if hours > 4.9 -> test accuracy 0.88

Always compare against a baseline: predicting the majority class already scores 60%. The 'model' here is a single learned threshold found only on training data, then judged on unseen test data (88%). The true rule (hours > 5 plus noise) is recovered closely — the noise is why 100% is impossible.

Common mistakes

Applications

Key takeaways

Going further