Distributed Tracing: Understanding and Instrumenting Spans
Explain the precise relationship between a trace and a span, describe how trace/span IDs propagate across service boundaries via headers like traceparent, and instrument meaningful child spans around database queries and external HTTP calls using PHP OpenTelemetry.
The 'Observability' lesson covered OpenTelemetry-based distributed tracing in PHP at an introductory level — enough to understand what tracing is for and how to get a basic trace flowing. This lesson goes deeper specifically on spans: what a trace and a span actually are and how they relate, exactly how IDs propagate across a service boundary, and — the part that matters most in practice — how to create spans around database queries and external calls that are actually useful for finding N+1 queries and slow APIs, rather than spans that just add noise.
What a Trace and a Span Actually Are
A trace represents the entire journey of one logical operation — almost always one incoming request — from the moment it enters your system to the moment a response is returned, potentially passing through many services along the way. A trace is identified by a single trace ID that stays constant across every service the request touches, which is exactly what lets a tracing backend (Jaeger, Tempo, Honeycomb, a vendor APM) reconstruct the full, cross-service picture of one request from pieces reported independently by each service.
A span represents one unit of work within that trace — a single function call, a single database query, a single outbound HTTP request, or a broader unit like 'handle the whole controller action'. Every span has its own span ID, a start time and duration, a name, and a set of key-value attributes describing it. Spans nest: a span can have a parent span, forming a tree (technically the OpenTelemetry data model allows more general parent-child relationships, but a tree is the common case) where the root span is usually the entire request, and children represent the sub-operations performed while handling it — the controller action span might have child spans for 'validate input', 'query database', and 'call payment API', and the database-query span might itself be a child of a broader 'process order' span.
This parent-child structure is what makes a trace visualization (a 'waterfall' or flame graph) so useful for debugging performance: the width of each span shows how long it took, its horizontal position shows when it started relative to its parent, and nested spans immediately reveal whether time was spent sequentially (one child span starts only after the previous one ends) or in parallel (children overlap), and which specific sub-operation — not just 'the request as a whole' — actually consumed the bulk of the total time.
• Trace = the whole request's journey across however many services it touches, identified by one trace ID. • Span = one unit of work within the trace (a query, an HTTP call, a function) with its own span ID, timing, name, and attributes. • Spans form a parent-child tree; the root span is usually the whole request, children are the sub-operations. • A trace waterfall/flame graph visualizes span nesting and timing, revealing exactly which sub-operation consumed the time.
Trace/Span ID Propagation Across Service Boundaries
Within a single process, propagating the current trace and span context is handled by the tracing SDK's in-memory context — each new span knows its parent because the SDK tracks 'the currently active span' as work executes. The harder problem is propagating that context across a network call to another service, so that a span created in Service B correctly attaches as a child of the span that made the call from Service A, rather than starting an unrelated, disconnected trace.
The standard mechanism for this is the W3C Trace Context specification, primarily the traceparent HTTP header. Its format is version-traceid-spanid-flags, for example 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01 — the trace ID (32 hex characters) stays constant across every service, the span ID (16 hex characters) identifies the specific span that made this particular outbound call (which becomes the parent span ID for whatever span the receiving service creates), and the flags byte communicates sampling decisions (whether this trace should actually be recorded in detail). When Service A makes an HTTP call to Service B, its OpenTelemetry SDK automatically injects a traceparent header carrying the current trace ID and the current span's ID; Service B's SDK, on receiving the request, extracts that header and creates its own root-of-service span as a child of the span ID it received, continuing the same trace rather than starting a new one.
This is why enabling auto-instrumentation for your HTTP client and your incoming-request framework layer matters so much in practice: as long as both ends understand traceparent (which every mainstream OpenTelemetry SDK does by default), trace continuity across service boundaries requires no manual header-passing code at all — it happens automatically as part of the instrumented HTTP client and server integration.
• traceparent header format: version-traceid-spanid-flags (e.g. 00-<32 hex trace id>-<16 hex span id>-01). • Trace ID stays constant across every service in the chain; each hop gets its own new span ID. • The span ID in an outgoing traceparent becomes the parent span ID for the span the receiving service creates. • OpenTelemetry SDKs inject/extract traceparent automatically via HTTP client/server auto-instrumentation — no manual header code needed in most cases.
Instrumenting Meaningful Spans for Database Queries and External Calls
Not every line of code deserves its own span — a trace with hundreds of tiny, low-value spans is as hard to read as no tracing at all. The two categories that consistently earn a dedicated child span are database queries and external service calls, because both are I/O operations whose latency is highly variable and whose timing, viewed in a waterfall, directly answers the two most common performance questions: 'is the database slow' and 'is a third-party API slow'.
Naming matters: a span named generically ('query' or 'http request') repeated hundreds of times in a trace is nearly useless. Convention (and OpenTelemetry semantic conventions) favor naming a database span after the operation and target, like SELECT orders or db.query orders, and naming an HTTP call span after the target and route pattern, like GET payment-service /charge — specific enough that scanning a list of span names in a trace immediately shows what was called, without needing to open each one.
Attributes attached to a span should describe what makes that call unique — the SQL statement (or a parameterized/normalized form of it, never with literal user data interpolated in, to avoid leaking PII into a tracing backend that may have weaker access controls than your primary database), the table name, the target host and route for an HTTP call, and the response status code — but never secrets, tokens, full request bodies, or raw personal data as attribute values. Done well, this instrumentation makes two extremely common performance bugs visible almost immediately in the waterfall view: an N+1 query pattern shows up as dozens of near-identical, sequential database-query spans with the same name and shape, one per row of an outer result set, where a single JOIN or eager-load would have produced one span; and a slow external API shows up as a single wide span whose duration dwarfs everything else in the trace, immediately pointing at the dependency rather than your own code as the source of latency.
• Reserve dedicated spans for I/O with variable, meaningful latency — database queries and external service calls are the two default candidates. • Name spans specifically (e.g. 'SELECT orders', 'GET payment-service /charge'), not generically ('query', 'http request'). • Attach the query/table or URL/route/status as attributes; never put secrets, tokens, or raw PII into attribute values. • N+1 queries appear as many near-identical sequential spans; a slow external API appears as one disproportionately wide span.
Code Examples
Every service in the chain that receives this header extracts the same trace ID (keeping the whole request's journey linked) and treats the span ID as the parent for whatever new span it creates, before generating its own fresh span ID to pass along on any further outbound calls it makes.
Each function creates its own child span (nested under whatever span was active when it was called, such as the request's root span), names it specifically after the operation/target, attaches attributes that describe the call without leaking literal parameter values or secrets, and always calls span->end() in a finally block so the span closes even when the call throws — an unclosed span is a common instrumentation bug that silently corrupts trace timing.
$span = $tracer->spanBuilder('SELECT orders')->setSpanKind(SpanKind::KIND_CLIENT)->startSpan();
Common mistake: Forgetting span->end() in a finally block means a span that throws never closes, which can leave its duration unbounded or missing entirely in the trace backend.
$span = $tracer->spanBuilder('query')->startSpan(); // same generic name for every single query in the app
$span = $tracer->spanBuilder('SELECT orders')->startSpan(); // named after the operation and table
A trace full of identically-named 'query' spans forces you to open every single one to see what it did; specific names make an N+1 pattern or a slow query visible just by scanning the span list.
$span->setAttribute('db.statement', "SELECT * FROM users WHERE email = '{$email}'");
$span->setAttribute('db.statement', 'SELECT * FROM users WHERE email = ?');
Interpolating literal parameter values into a span attribute can leak PII (like an email address) into the tracing backend, which often has weaker access controls than the primary database — always record the parameterized/normalized query form.
$span->end(); // called only on the success path try {... } finally { $span->end(); }
If the code between startSpan() and end() throws and end() is only called on the success path, the span never closes — always end a span in a finally block so it closes on every code path, including exceptions.
What is the relationship between a trace and a span? A. They are two names for the same concept B. A trace is the entire request's journey; a span is one unit of work within that trace, and spans can nest as parent/child C. A span always spans multiple services, while a trace is confined to one service D. A trace can only contain exactly one span
B. A trace is the entire request's journey; a span is one unit of work within that trace, and spans can nest as parent/child — A trace (identified by one trace ID) represents the whole request's journey, potentially across many services; a span represents one unit of work within it and has its own span ID, with spans nesting in parent-child relationships.
When Service A calls Service B over HTTP with OpenTelemetry instrumentation enabled, how does Service B know which trace to attach its new span to? A. It generates a brand new trace ID every time, unrelated to Service A B. It reads the trace ID and parent span ID from the incoming traceparent header C. It queries a shared database table for the current trace ID D. Trace continuity across services is not possible in PHP
B. It reads the trace ID and parent span ID from the incoming traceparent header — Service A's SDK injects a traceparent header carrying the current trace ID and its own span ID; Service B's SDK extracts that header and creates its span as a child of the received span ID, continuing the same trace.
In a trace waterfall view, what does a pattern of dozens of near-identical, sequential, same-named database-query spans typically indicate? A. A slow external API B. An N+1 query problem, where a query is being re-run once per row of an outer result set instead of being batched C. A misconfigured trace sampler D. The database connection pool is exhausted
B. An N+1 query problem, where a query is being re-run once per row of an outer result set instead of being batched — Many nearly identical, sequential query spans with the same shape is the classic visual signature of an N+1 query pattern — a single JOIN or eager-load would have produced one span instead of one per row.
Why should a span attribute never contain a literal SQL parameter value like a user's email address? A. It would make the span name too long B. OpenTelemetry attributes only accept numeric values C. It can leak PII into the tracing backend, which often has weaker access controls than the primary database D. It would break traceparent propagation
C. It can leak PII into the tracing backend, which often has weaker access controls than the primary database — Tracing backends are frequently more broadly accessible than the primary database, so recording literal parameter values as span attributes risks leaking sensitive data outside its normal access boundary — record the parameterized/normalized query instead.
traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
│ └────────── trace id (32 hex) ─────────┘ └ span id (16 hex) ┘ └flags┘
version<?php
declare(strict_types=1);
use OpenTelemetry\API\Trace\SpanKind;
use OpenTelemetry\API\Trace\StatusCode;
$tracer = $tracerProvider->getTracer('order-service');
function fetchOrder(PDO $pdo, $tracer, int $orderId): array
{
$span = $tracer->spanBuilder('SELECT orders')
->setSpanKind(SpanKind::KIND_CLIENT)
->startSpan();
$scope = $span->activate();
try {
$span->setAttribute('db.system', 'mysql');
$span->setAttribute('db.sql.table', 'orders');
$span->setAttribute('db.statement', 'SELECT * FROM orders WHERE id = ?'); // parameterized, no literal value
$stmt = $pdo->prepare('SELECT * FROM orders WHERE id = ?');
$stmt->execute([$orderId]);
$row = $stmt->fetch(PDO::FETCH_ASSOC) ?: [];
$span->setStatus(StatusCode::STATUS_OK);
return $row;
} catch (\Throwable $e) {
$span->recordException($e);
$span->setStatus(StatusCode::STATUS_ERROR, $e->getMessage());
throw $e;
} finally {
$span->end();
$scope->detach();
}
}
function chargePayment($tracer, HttpClientInterface $http, array $payload): array
{
$span = $tracer->spanBuilder('POST payment-
…