The Metrics Catalog Every Production PHP App Should Expose

Enumerate the full set of metrics a production PHP application should expose beyond the four golden signals, explain why each matters and how it's typically collected, and instrument a request-latency histogram using a PHP Prometheus client.

The 'Observability' lesson introduced the four golden signals (latency, traffic, errors, saturation) at a high level. This lesson builds out the full, concrete metrics catalog a real PHP application should expose in production — the specific counters, histograms, and gauges teams actually wire up, why each one matters operationally, and how they're typically collected with StatsD or Prometheus client libraries. It ends with a working example of emitting a request-latency histogram around a request's lifecycle.

Request-Level Metrics: Count, Latency Percentiles, Error Rate, Throughput

Request count is the simplest metric: a counter incremented once per handled request, usually labeled by route and HTTP method so it can be broken down (GET /api/orders vs POST /api/orders). On its own it answers 'how much traffic are we getting', but its real value comes from being the denominator for error rate and the numerator for throughput.

Request latency should always be recorded as a histogram (or summary), never as a single average. An average latency of 200ms can hide a distribution where 99% of requests finish in 50ms and 1% take 15 seconds — the average looks fine while a meaningful slice of real users have a terrible experience. Percentiles fix this: p50 (median) shows the typical experience, p95 shows what the slower-than-typical 5% of users see, and p99 shows the tail — the requests most likely to be timeouts, retries, or abandoned sessions. Most teams alert on p95/p99, not on the average, because the average is the metric least likely to reflect what any actual user experienced.

Error rate is the ratio of failed requests (typically 5xx responses, or exceptions that escaped the request handler) to total requests, usually expressed as a percentage over a rolling window. It should be tracked per route/endpoint, not just globally, because a single broken endpoint can be masked by a healthy overall average across a large app. Throughput is requests-per-second (or per-minute), and matters both on its own — sudden drops often mean an upstream problem, like a load balancer misrouting traffic, before your error rate even moves — and as context for interpreting the other three: high latency at low throughput is a different problem than high latency at peak throughput.

• Request count — counter, labeled by route + method; the traffic signal and the denominator for rates. • Request latency — ALWAYS a histogram; report p50/p95/p99, never just the average, since averages hide tail latency. • Error rate — failed-to-total ratio per route, not just globally, so one broken endpoint isn't hidden. • Throughput — requests/sec; sudden drops can reveal upstream problems before error rate moves.

Background Work: Queue Depth and Database Query Latency

Queue depth is the number of jobs currently waiting in a background job queue (Redis-backed queues like Laravel's, RabbitMQ, SQS, Beanstalkd) and is one of the most important early-warning metrics an application can expose, because it's a direct measure of whether workers are keeping up with the rate jobs are being enqueued. A queue depth that's flat or slowly draining is healthy; a queue depth that's climbing steadily means either workers are too slow, too few, or something downstream (a database, an external API) that jobs depend on has degraded — and it will keep climbing, invisibly, until someone notices delayed emails, delayed notifications, or a queue so backed up it takes hours to drain even after the root cause is fixed. It's usually collected as a gauge, sampled periodically (every few seconds) directly from the queue backend (e.g. LLEN on a Redis list, or a queue-specific API call).

Database query latency is tracked as a histogram, similarly to request latency and for the same reason — averages hide the slow-query tail that's usually the actual cause of a slow endpoint. It's commonly instrumented by wrapping the PDO/query-execution layer once (a query logger, an event listener, or a decorator around your database abstraction) rather than annotating every individual query call site, and is often labeled by query type or table so a specific slow table or query pattern can be identified quickly. This metric is frequently the fastest way to root-cause a latency regression, because 'is it the database or is it us' is one of the first questions any on-call engineer asks when p99 spikes.

• Queue depth — gauge, sampled from the queue backend; a steadily climbing depth is an early warning that workers can't keep up. • DB query latency — histogram, wrapped once at the query-execution layer, labeled by query/table when possible. • Both metrics exist specifically to answer 'is it us, or is it something we depend on' quickly during an incident.

Cache Hit Rate and Resource Usage (CPU/Memory)

Cache hit rate is the percentage of cache lookups (Redis, Memcached, an in-process cache) that find a value versus miss and fall through to a slower path (usually the database). It's typically computed from two counters — a hits counter and a misses counter, incremented at every cache read — divided at query time in the metrics dashboard rather than computed in application code. A cache hit rate that drops suddenly is a strong, fast signal that something changed: a deploy that altered cache keys, a cache eviction from memory pressure, or a cold cache after a restart — and because cache misses fall through to the database, a falling hit rate often shows up moments later as rising database latency, making the two metrics worth viewing side by side.

CPU and memory usage are host/process-level resource metrics, usually collected by an infrastructure-level agent (Node Exporter, the platform's own container metrics, or PHP-FPM's own status page for process-level detail) rather than instrumented by hand inside application code. For a PHP-FPM deployment specifically, memory per worker process and the count of busy vs idle workers are often more actionable than raw host CPU, because PHP-FPM has a fixed pool size (pm.max_children) — a pool that's fully busy under load is a very direct explanation for rising latency and rejected connections, and is exactly the kind of saturation signal the four golden signals refer to. These are collected via StatsD (metrics pushed to a local StatsD daemon, which batches and forwards them) or a Prometheus client library that exposes a /metrics HTTP endpoint for a Prometheus server to scrape on an interval.

• Cache hit rate — hits / (hits + misses); a sudden drop is a fast signal of a cache problem, often followed by rising DB latency. • CPU/memory — usually collected at the infra/process level (Node Exporter, PHP-FPM status), not hand-instrumented per request. • For PHP-FPM specifically, busy-vs-idle worker count against pm.max_children is often more actionable than raw CPU. • Collection mechanism: StatsD (push to a local daemon) or a Prometheus client library exposing /metrics for scraping.

Code Examples

The histogram buckets let Prometheus compute p50/p95/p99 later at query time (via histogram_quantile in PromQL) rather than forcing the application to pre-compute a single percentile; labeling by route/method/status lets the same metric be sliced per-endpoint or aggregated app-wide, and pairing it with a matching counter gives both the latency distribution and the raw request/error counts from one instrumentation pass.

$histogram->observe($duration, [$route, $method, (string) $status]);

Queue depth is a gauge (a value that can go up or down), not a counter, and is sampled on an interval from the queue backend rather than incremented inline with request-handling code — it reflects the current state of a shared resource, not an event count.

// dashboard shows: avg(http_request_duration_seconds) // dashboard shows: histogram_quantile(0.95, http_request_duration_seconds_bucket) and p99

Averaging latency hides tail behavior entirely — a handful of very slow requests can be invisible in an average while still ruining the experience for the users who hit them. Percentiles from a histogram reveal what averages hide.

// only a global error-rate metric, no per-route breakdown // error rate labeled by route, e.g. http_requests_total{status=~"5.."} / http_requests_total, grouped by route

A single badly broken endpoint can be statistically invisible in an app-wide average error rate if the rest of the app is healthy and high-traffic; per-route labels are what actually surface it.

Why should request latency be tracked as a histogram rather than a simple average? A. Histograms use less memory than a single number B. An average can look healthy even when a meaningful percentage of requests are very slow; percentiles (p50/p95/p99) reveal that tail C. Prometheus does not support averages at all D. Histograms are required by the four golden signals specification

B. An average can look healthy even when a meaningful percentage of requests are very slow; percentiles (p50/p95/p99) reveal that tail — Averages hide distribution shape — a histogram lets you compute percentiles later, and p95/p99 are what typically matter for user experience and alerting, since they capture the slow tail an average smooths over.

A queue's depth metric has been climbing steadily for the last hour. What does this most directly indicate? A. The application has a memory leak B. The cache hit rate has improved C. Workers are not keeping up with the rate jobs are being enqueued D. The database has run out of disk space

C. Workers are not keeping up with the rate jobs are being enqueued — A steadily climbing queue depth is a direct signal that consumption (worker throughput) is falling behind production (enqueue rate) — the cause could be too few workers, slow workers, or a degraded downstream dependency, but the depth metric itself is what surfaces the imbalance early.

How is cache hit rate typically computed? A. As a single directly-reported percentage counter B. From two counters, hits and misses, divided at query time in the dashboard C. By sampling CPU usage during cache reads D. It cannot be measured in PHP applications

B. From two counters, hits and misses, divided at query time in the dashboard — Applications increment separate hits and misses counters on every cache read; the ratio (hit rate) is computed later at query/dashboard time, which also lets you view raw counts and the rate together.

<?php
declare(strict_types=1);

use Prometheus\CollectorRegistry;
use Prometheus\Storage\Redis as PrometheusRedis;

PrometheusRedis::setDefaultOptions(['host' => 'redis', 'port' => 6379]);
$registry = new CollectorRegistry(new PrometheusRedis());

$histogram = $registry->getOrRegisterHistogram(
    'app',                       // namespace
    'http_request_duration_seconds',
    'Duration of HTTP requests in seconds',
    ['route', 'method', 'status'], // labels
    [0.01, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10] // histogram buckets, in seconds
);

$requestCounter = $registry->getOrRegisterCounter(
    'app',
    'http_requests_total',
    'Total HTTP requests handled',
    ['route', 'method', 'status']
);

$start = microtime(true);

// ... the actual request is handled here by the application/router ...
$route = '/api/orders/{id}';
$method = $_SERVER['REQUEST_METHOD'];
$status = http_response_code() ?: 200;

$duration = microtime(true) - $start;

$histogram->observe($duration, [$route, $method, (string) $status]);
$requestCounter->inc([$route, $method, (string) $status]);

// A separate /metrics endpoint exposes everything for Prometheus to scrape:
// $renderer = new Prometheus\Rend
…

<?php
$queueDepth = $registry->getOrRegisterGauge(
    'app',
    'job_queue_depth',
    'Number of jobs waiting in the queue',
    ['queue']
);

// Run periodically (e.g. from a scheduled task every 15s), not per-request:
foreach (['default', 'emails', 'reports'] as $queueName) {
    $depth = $redis->llen("queue:{$queueName}");
    $queueDepth->set($depth, [$queueName]);
}