Generative AI Applications: Prompting, RAG & Agents

Building useful products on foundation models. Learn prompt engineering (instructions, examples, structure, chain-of-thought, output formats), embeddings and semantic similarity, retrieval-augmented generation (chunking, embedding, vector databases, retrieval, re-ranking, grounding and citations), TF-IDF/BM25 vs dense retrieval, tool use and function calling, agents and planning, fine-tuning vs RAG, evaluation of GenAI apps, guardrails, cost and latency — with a runnable TF-IDF retriever that selects context for a prompt.

Most generative AI products don't train models — they combine a foundation model with good prompts, company data retrieved at query time, and tools the model can call. This is where AI engineering creates most value today.

Retrieval-augmented generation (RAG) grounds answers in trusted documents, reducing hallucinations and keeping knowledge up to date without retraining.

An analogy

RAG is like an open-book exam: instead of relying on memory (which can be wrong or outdated), the model is handed the most relevant pages from the textbook right before answering — and can cite them.

Prompt engineering

Principles of good prompts:

The RAG pipeline

Offline: load documents → clean → chunk (e.g. 300–800 tokens with overlap) → embed → store in a vector database (FAISS, pgvector, Pinecone, Weaviate) with metadata. Online: embed query → retrieve top-k (hybrid dense + keyword) → optionally re-rank → build prompt with retrieved chunks → generate answer with citations → log for evaluation. Chunking strategy, metadata filters and re-ranking often matter more than the LLM choice.

Tools and agents

Function calling lets models output structured tool requests (search, database query, calculator, send email) that your code executes and returns. Agents loop: plan → act with tools → observe → repeat, for multi-step tasks (research, coding, data analysis). Protocols like MCP standardise tool connections. Agents need guardrails: permission checks, sandboxing, step limits and human approval for irreversible actions.

Choosing techniques and operating GenAI

Prompting first; RAG when knowledge is private or changing; fine-tuning for style, format or narrow skills (often with LoRA); combinations are common. Evaluate with task datasets (faithfulness, answer relevance, context recall), LLM-as-judge and human review. Guardrails: input filtering, prompt-injection defences (treat retrieved text as data, not instructions), output validation, PII redaction. Optimise cost/latency with smaller models, caching, batching, streaming and routing.

Formulas

Worked example: Cosine similarity

Given: a = (1, 2, 0), b = (2, 1, 1)

Find: cos.

Answer: 0.730

Worked example: Chunk count

Given: 200-page manual ≈ 100 000 tokens, 500-token chunks, 50 overlap

Find: Chunks.

Answer: ≈ 223 chunks

Worked example: Query cost

Given: 5 chunks × 500 tokens + 300 prompt, 400 output; $3/M in, $15/M out

Find: Cost per query.

Answer: $0.0144

A tiny RAG retriever with TF-IDF

import math
from collections import Counter
# Retrieval for RAG: TF-IDF vectors + cosine similarity over a tiny knowledge base
docs = {
    "leave": "employees get 24 days of paid leave per year and leave requests go to the manager",
    "expense": "travel expenses must be submitted within 30 days with receipts for reimbursement",
    "laptop": "a new laptop is issued every three years and lost laptops must be reported to IT",
}
def tok(s): return s.lower().replace("?", "").split()
N = len(docs)
df = Counter(w for text in docs.values() for w in set(tok(text)))
def vec(text):
    tf = Counter(tok(text))
    return {w: c * math.log((1 + N) / (1 + df.get(w, 0))) for w, c in tf.items()}
def cos(a, b):
    num = sum(a[w] * b.get(w, 0) for w in a)
    den = math.sqrt(sum(x * x for x in a.values())) * math.sqrt(sum(x * x for x in b.values()))
    return num / den if den else 0.0
query = "how many days of leave do I get?"
qv = vec(query)
ranked = sorted(((cos(qv, vec(t)), k) for k, t in docs.items()), reverse=True)
for s, k in ranked:
    print(f"{k:8s} similarity {s:.3f}")
top = ranked[0][1]
print("prompt -> Answer using only this context: " + docs[top][:45] + "... + question")
# leave    similarity 0.242
# expense  similarity 0.013
# laptop   similarity 0.000
# prompt -> Answer using only this context: employees get 24 days of paid leave per year ... + question

The retriever correctly picks the leave policy — 'leave' is a rare, informative word (high IDF), while 'days' appears in two documents and adds less. Production systems use dense embedding models (which also match synonyms like 'vacation') plus BM25 hybrid search and re-rankers, then pass the top chunks into the prompt with instructions to answer only from them and cite sources.

Common mistakes

Applications

Key takeaways

Going further