Liveness vs Readiness: Building Health Checks That Don't Lie

Distinguish liveness from readiness probes precisely, explain why conflating them causes outages, and implement a real PHP readiness endpoint that checks actual dependencies with sensible timeouts.

The Observability module briefly mentioned health, readiness, and liveness checks as part of the broader observability toolkit. This lesson goes much deeper on that one topic specifically: the precise semantic difference between a liveness check and a readiness check, why treating them as interchangeable is a common cause of self-inflicted outages, and how to build a readiness endpoint that actually verifies your dependencies instead of just returning a static 200 OK.

Liveness vs Readiness: Two Different Questions

A liveness probe answers one narrow question: 'is this process still alive and functioning, or is it stuck/deadlocked/corrupted such that it should be killed and restarted?' It should check almost nothing beyond 'can the process respond to a request at all' — a liveness check that does too much work, or checks too much external state, risks reporting the process as dead simply because something it depends on is having a bad day, which is exactly the wrong signal to act on.

A readiness probe answers a completely different question: 'is this specific instance, right now, in a state where it should receive traffic?' A process can be perfectly alive (not deadlocked, responsive, healthy on its own terms) while still not being ready — for example, during startup before it has finished warming a cache, or at any point when a required downstream dependency (database, cache, queue) is unreachable or too slow. Readiness failing means 'temporarily stop sending this instance traffic'; it does not mean 'this process is broken and should be killed'.

The orchestrator or load balancer acts very differently on each signal: a failed liveness probe typically triggers a restart of the process/container; a failed readiness probe typically just removes that instance from the load-balancing pool until it reports ready again, without restarting anything. This is the entire reason the two checks must be kept semantically distinct rather than treated as two names for the same 'health' concept.

• Liveness: 'is the process alive/functioning, or should it be killed and restarted?' — checks almost nothing external. • Readiness: 'should this instance receive traffic right now?' — checks actual dependencies. • Failed liveness → restart the process. Failed readiness → pull the instance from the traffic pool, no restart. • A process can be alive but not ready (e.g. still warming up, or a dependency is down/slow).

Why Conflating Them Causes Outages

The classic failure mode: a team implements a single /health endpoint that checks the database, wires both the liveness AND readiness probes to it, and calls it done. Then the database gets slow — not down, just slow, perhaps under a heavy reporting query or a brief network blip. Every instance's combined health check starts timing out or failing. Because it's wired to liveness, the orchestrator concludes every instance is dead and restarts all of them simultaneously.

This makes an already-bad situation catastrophically worse in two ways: first, a slow database is not a reason to kill and restart application processes — the processes themselves are fine, and restarting them does nothing to fix the database, while consuming CPU/memory on cold starts and dropping in-flight requests. Second, restarting every instance at once removes the entire fleet from service simultaneously, so instead of degraded performance (which is what a slow-but-not-down database should cause) you get a full outage where zero instances are available to serve any traffic while they all restart and reconnect at once — often re-triggering the same database load spike that caused the initial slowness, in a feedback loop.

The correct design: liveness checks only the process's own basic responsiveness (can it accept a request and return something, is its event loop or worker not wedged) and never calls out to the database, cache, or any network dependency. Readiness checks the actual dependencies, and a failed readiness check simply and safely removes that one instance from the load balancer's rotation — traffic routes to the remaining ready instances, and the affected instance rejoins automatically the moment its dependencies recover, with no restart and no dropped in-flight work.

• Wiring both liveness and readiness to the same DB-checking endpoint is the classic mistake. • A slow (not down) dependency should never cause a restart — restarting doesn't fix the dependency and drops in-flight work. • Mass-restarting the whole fleet on a shared dependency blip turns 'degraded' into 'total outage'. • Correct design: liveness checks only the process itself; readiness checks dependencies and just adjusts traffic routing.

Building a Real Readiness Endpoint

A meaningful readiness endpoint checks each dependency the request path actually needs to succeed: can it open a database connection (or reuse a pooled one) and run a trivial query, can it reach the cache (e.g. a Redis PING), can it reach the message queue it publishes to. Each of these checks needs its own short timeout, deliberately shorter than the request timeout a real user would experience — the entire point of a readiness check is to fail fast and cleanly if something is wrong, and a health check that itself hangs for 30 seconds waiting on a dead connection defeats the purpose by blocking the orchestrator's own probe loop.

The response should reflect a clear, unambiguous status: an HTTP 200 when every checked dependency is healthy, and a non-2xx status (409 or 503 are common conventions) when any required dependency fails, ideally with a small JSON body breaking down which specific check failed — this detail is invaluable during an incident, since it tells the on-call engineer immediately whether it's the database, cache, or queue that's the problem, without them needing to dig through logs first.

It's worth explicitly deciding, per dependency, whether its failure should actually fail readiness at all. A cache is often an optimization, not a hard requirement — if your application degrades gracefully without it (slower responses, but still correct ones), it may be better to log a warning and stay ready rather than pull the instance from rotation over a non-critical dependency. A primary database that the request path cannot function without should, by contrast, fail readiness immediately.

• Check each real dependency (DB, cache, queue) with a short timeout of its own, well under the normal request timeout. • Return 200 when fully healthy; a non-2xx status (409/503) with a breakdown when something fails. • Decide per-dependency whether its failure should actually fail readiness — hard requirement vs graceful degradation. • A hanging health check is worse than a failing one — always bound every check with a timeout.

Code Examples

Every dependency check has its own short timeout (1 second for the DB connection, 0.5 seconds for Redis and the queue socket) so a completely dead dependency fails this endpoint quickly instead of hanging it. The JSON body breaks down exactly which check failed, and the HTTP status (200 vs 503) is what a load balancer or orchestrator actually reads to decide whether to route traffic here. Expected result: {"status":"not_ready","checks":{"database":true,"redis":false,"queue":true}} http_response_code($allHealthy ? 200: 503);

This endpoint deliberately checks nothing external. If it can execute at all, the process is alive; whether it's ready to serve real traffic is a completely separate question answered by the readiness endpoint above. Wiring an orchestrator's restart-on-failure liveness probe to this trivial endpoint means a slow database never triggers an unnecessary restart of a perfectly healthy process.

Pointing both the liveness AND readiness probes at the same /health endpoint that checks the database

Liveness checks only the process itself (no external calls); readiness checks the database/cache/queue separately.

A slow or briefly unreachable database should reduce traffic to affected instances (readiness), not trigger a fleet-wide restart (liveness) — conflating the two turns a degraded dependency into a full outage.

A health check that opens a DB connection with no timeout at all

Set an explicit short connect/query timeout (e.g. PDO::ATTR_TIMEOUT => 1) on every dependency check.

An unbounded health check can hang as long as the dependency hangs, which can exceed the orchestrator's own probe timeout and produce confusing, delayed failure signals instead of a fast, clear one.

What question does a readiness probe answer, as opposed to a liveness probe? A. Whether the process should be killed and restarted B. Whether this specific instance should currently receive traffic C. Whether the PHP version is up to date D. Whether the code has any syntax errors

B. Whether this specific instance should currently receive traffic — Readiness answers 'should this instance get traffic right now', which can be false even while the process itself is perfectly alive (e.g. a dependency is temporarily unreachable).

Why does wiring both liveness and readiness to the same dependency-checking endpoint risk causing an outage? A. It has no risk; they are meant to be identical B. A slow/unreachable dependency would then trigger a fleet-wide restart instead of just removing affected instances from traffic C. It makes the endpoint respond faster D. It is only a style preference with no operational impact

B. A slow/unreachable dependency would then trigger a fleet-wide restart instead of just removing affected instances from traffic — If liveness is tied to a dependency check, a shared dependency blip can make every instance appear 'dead' simultaneously, causing a mass restart that turns a degraded dependency into a full outage.

Why must each dependency check inside a readiness endpoint have its own short timeout? A. To make the JSON response smaller B. So a single hung or dead dependency can't cause the health check itself to hang, delaying a clear pass/fail signal C. PHP requires a timeout on every function call D. Timeouts are only needed for liveness checks, never readiness

B. So a single hung or dead dependency can't cause the health check itself to hang, delaying a clear pass/fail signal — Without a short, explicit timeout, a stuck dependency can make the health check hang indefinitely, defeating its purpose of giving a fast, unambiguous status.

<?php
declare(strict_types=1);

header('Content-Type: application/json');

function checkDatabase(PDO $pdo): bool {
    try {
        $stmt = $pdo->query('SELECT 1');
        return $stmt !== false;
    } catch (\Throwable $e) {
        return false;
    }
}

function checkRedis(string $host, int $port, float $timeoutSeconds = 0.5): bool {
    $conn = @fsockopen($host, $port, $errno, $errstr, $timeoutSeconds);
    if ($conn === false) {
        return false;
    }
    stream_set_timeout($conn, (int) $timeoutSeconds);
    fwrite($conn, "PING\r\n");
    $response = fgets($conn, 128);
    fclose($conn);
    return $response !== false && str_contains($response, 'PONG');
}

function checkQueue(string $host, int $port, float $timeoutSeconds = 0.5): bool {
    $conn = @fsockopen($host, $port, $errno, $errstr, $timeoutSeconds);
    if ($conn === false) {
        return false;
    }
    fclose($conn);
    return true;
}

// Set a hard cap on the DB check's own connection timeout, separate
// from the PDO instance's default, so a stuck DB can't hang this endpoint.
$dbPdo = new PDO(
    'mysql:host=127.0.0.1;dbname=app',
    getenv('DB_USER'),
    getenv('DB_PASS'),
    [PDO::ATTR_TIMEOUT => 
…

<?php
declare(strict_types=1);

// Liveness: only proves the PHP process can accept and answer a request.
// Never checks the database, cache, or any network dependency here.
header('Content-Type: application/json');
http_response_code(200);
echo json_encode(['status' => 'alive']);