Mutation Testing: Measuring How Good Your Tests Actually Are
Understand what Infection does (introducing deliberate small code mutations and checking whether tests catch them), the common mutation operators, how mutation score is calculated and what a surviving mutant indicates, why high line coverage can coexist with weak tests, and how to read and act on a mutation testing report.
Line coverage answers 'did any test execute this line' — it says nothing about whether a test would notice if that line's behavior were wrong. A test that calls a function and asserts nothing about its return value achieves 100% coverage on every line of that function while catching zero bugs in it. Mutation testing exists to measure the question coverage can't: it takes your actual source code, introduces a small deliberate bug into a copy of it (flip a > to >=, change + to -, delete a method call), reruns your test suite against that mutated copy, and checks whether any test fails. If a test fails, the mutant was 'killed' — your tests actually noticed the bug. If every test still passes against the mutated code, the mutant 'survived' — and a surviving mutant is direct, concrete evidence that a real bug in that exact spot would also slip through your tests undetected.
Infection: the PHP mutation testing framework
Infection is the standard PHP mutation testing tool. Given a test suite (PHPUnit or Pest) and a source directory, it works in three phases: first it runs the full test suite once, unmutated, to establish a baseline of which tests cover which lines (using your existing coverage driver, Xdebug or PCOV). Second, for every mutable location in the covered code, it generates a mutant — a copy of that one file with a single small change applied — and reruns only the subset of tests that the baseline coverage data says actually exercise that specific line (not the whole suite every time, which is what makes mutation testing computationally feasible on a real codebase rather than combinatorially explosive). Third, it aggregates the result of every mutant into a report: killed, survived, or a few other outcomes covered below.
Running it is infection --threads=4 (parallelized, since a codebase can generate thousands of mutants and each requires a test run), typically configured via infection.json5 to set minimum acceptable mutation score thresholds, which paths to mutate, and which specific mutators to enable or disable.
Mutation operators: what kinds of changes Infection introduces
A mutation operator is a rule describing one specific kind of small, mechanical code change. Infection ships dozens, grouped by category. Conditional-boundary mutators flip comparison operators at their boundary — > becomes >=, < becomes <=, === becomes !== — because off-by-one boundary bugs are exactly where real-world logic errors cluster. Arithmetic mutators swap operators — + becomes -, * becomes / — to check whether a test would notice a wrong calculation. Logical mutators flip && to || and vice versa, and negate boolean conditions entirely (if ($x) becomes if (!$x)). Return-value mutators replace a method's return value with a boundary-ish alternative (a non-empty array becomes empty, a truthy scalar becomes falsy, an object becomes null where nullable). Removal mutators delete a method call entirely (particularly effective at catching tests that check a return value but never verify a required side effect actually happened, like a logger call or an event dispatch).
Each mutator represents a hypothesis about a realistic class of bug a developer might actually introduce by accident — a mutation operator isn't trying to produce exotic, unrealistic code; it's modeling the everyday off-by-one, wrong-operator, forgot-to-call-this mistakes that happen in real pull requests, which is exactly why a test suite's ability to catch them is a meaningful signal about real-world bug-catching ability.
Mutation score and what it means
Mutation Score = (killed mutants) / (total mutants) x 100, restricted to mutants Infection actually generated on covered code — a mutant introduced in a line no test executes at all is reported separately as 'not covered' rather than counted as survived, since 'no test even runs this line' is a coverage gap, a distinct problem from 'a test runs this line but wouldn't notice if it broke.' A Mutation Score Indicator (MSI) of 80% means 80% of the mutants Infection was able to generate and test were caught by some test failing; the remaining 20% represent either surviving mutants or mutants Infection judged equivalent to the original (a mutation that provably cannot change observable behavior — rare, but it happens, e.g. mutating dead code) and excludes from the score.
There's no universal 'good' mutation score threshold the way there sometimes is a rough consensus around coverage percentages — a payment-processing module reasonably targets 90%+ MSI given the cost of a bug there, while a thin presentation-layer wrapper with little logic of its own may not be worth chasing above 60-70%. What matters more than the absolute number is the trend (is MSI moving up or down as the codebase grows) and which specific mutants survive (a survived mutant in critical business logic is a much bigger deal than one in a getter).
Surviving mutants: what they actually indicate
A surviving mutant means: this exact line was changed, in this exact small way, and every test in the suite still passed. That is direct evidence — not a proxy metric, an actual demonstrated fact — that if a developer introduced a real bug matching that mutation (flipped a boundary, swapped an operator) tomorrow, the test suite would not catch it before it shipped. This is the single biggest practical value mutation testing has over coverage: coverage tells you a line was *touched*, a surviving mutant tells you a line was touched by a test that *would not have noticed if that exact line were wrong*.
Reading a surviving-mutant report is a triage exercise, not a mechanical 'must reach zero' target. Some survivors point at a genuinely missing assertion — a test calls calculateDiscount() and checks nothing about the return value, so any arithmetic mutator there survives trivially, and the fix is adding the missing assertion. Others point at intentionally loose logic where the exact boundary genuinely doesn't matter for correctness (a >= 0 vs > -1 on some display-only rounding, say), where the mutant surviving is fine and not worth chasing. The report is a prioritized list of 'here's exactly where your tests wouldn't notice a bug' — the judgment of whether each one matters still belongs to a human.
Weak tests vs strong tests: coverage is necessary, not sufficient
A weak test achieves high line coverage while asserting little or nothing about behavior — calling a function and checking only that it didn't throw, or asserting on a return type without checking the value, or testing only the happy path of a function with three branches while all three lines still count as 'covered' because the happy-path test happens to execute all of them. Such a test suite can legitimately show 95%+ line coverage on a codebase with a mutation score in the 30-40% range — a large, real, and common gap that coverage alone will never reveal, because coverage instrumentation only tracks which lines executed, never what the test actually verified about the outcome.
A strong test asserts on the specific, meaningful outcome a change to that code would alter — not just 'the function returned something' but 'the function returned exactly this value for this input, and this different value for that edge-case input.' Mutation testing is effectively an automated way to audit assertion quality at scale: every surviving mutant is a concrete instance of 'a test executed this code but didn't check the thing that would have revealed a bug here,' which is a far more actionable signal than a coverage percentage for deciding where to actually invest testing effort.
• High coverage + low mutation score = tests execute code but don't verify outcomes (weak tests) • High coverage + high mutation score = tests both execute code and verify meaningful outcomes (strong tests) • Low coverage always implies a low mutation score ceiling — you can't kill a mutant on a line no test runs at all • 100% coverage is a floor worth having, never a proxy for 'well-tested'
Interpreting and acting on a mutation testing report in practice
In CI, Infection is typically run with a --min-msi threshold (fail the build if the overall mutation score indicator drops below it) and often a separate, stricter --min-covered-msi threshold (mutation score restricted to lines that do have some coverage, which isolates 'weak assertions' from 'no coverage at all' as two different failure modes). A common adoption pattern mirrors PHPStan's baseline approach: run Infection once on the existing codebase, record the current MSI as the enforced floor, then require that floor to never regress on new pull requests — new code has to meet the bar, existing code is paid down opportunistically rather than all at once.
Because mutation testing reruns the (targeted subset of the) test suite once per mutant, it is meaningfully slower than a normal test run or a coverage report — running it on every commit locally is usually impractical. The common workflow is running it in CI on pull requests (often only against changed files/lines via --filter or git-diff-based scoping, rather than the whole codebase every time) and running a full, unscoped pass periodically (nightly, or before a release) to catch drift across the whole codebase.
infection.json5: thresholds and mutator configuration
minMsi is the floor for the overall score; minCoveredMsi is stricter and isolates weak-assertion problems from no-coverage-at-all problems. Disabling a specific mutator is a deliberate exception, not a default — most teams start with @default: true and only exclude a mutator that's producing consistently non-actionable noise for their specific codebase.
{
"source": {
"directories": ["src"]
},
"logs": {
"text": "infection.log",
"summary": "summary.log"
},
"mutators": {
"@default": true,
"ArrayItemRemoval": false // disable a specific noisy mutator if needed
},
"minMsi": 75,
"minCoveredMsi": 85,
"testFramework": "phpunit"
}Code Examples
Both branches execute, so line coverage reports 100% for calculate(). But mutating >= to >, or 0.8 to 0.9, or 0.95 to 1.0, would not fail this test at all — assertIsFloat() passes regardless of the actual numeric value, so every arithmetic and boundary mutator on this method survives. The test proves the function returns *a* float; it proves nothing about whether the discount math is correct.
$this->assertIsFloat($calc->calculate(100.0, 10)); // hits the >= 5 branch
Common mistake: Asserting only on type (assertIsFloat) instead of on the actual expected value is the single most common source of surviving mutants in an otherwise 'well-covered' codebase.
Testing exactly at the boundary (5, the last year that still qualifies, and 4, the first year that doesn't) with an exact expected numeric value kills the ConditionalBoundary mutator (flipping >= to >) directly: if that mutation were applied, calculate(100.0, 5) would fall through to the 0.95 branch and return 95.0 instead of the asserted 80.0, failing the test. Asserting the precise expected value (assertSame(80.0, ...) rather than assertIsFloat) also kills any arithmetic mutator that alters 0.8 or 0.95. Expected result: Both tests pass against the original code; both tests FAIL against a mutant with >= flipped to > or with 0.8 changed to 0.9 — those mutants are killed instead of surviving.
<?php
declare(strict_types=1);
final class DiscountCalculator
{
public function calculate(float $price, int $loyaltyYears): float
{
if ($loyaltyYears >= 5) {
return $price * 0.8; // 20% off
}
return $price * 0.95; // 5% off
}
}
// --- Weak test: executes both branches (100% line coverage) but asserts
// almost nothing about the actual computed values. ---
final class DiscountCalculatorWeakTest extends \PHPUnit\Framework\TestCase
{
public function testCalculateReturnsAFloat(): void
{
$calc = new DiscountCalculator();
$this->assertIsFloat($calc->calculate(100.0, 10)); // hits the >= 5 branch
$this->assertIsFloat($calc->calculate(100.0, 1)); // hits the else branch
}
}<?php
declare(strict_types=1);
final class DiscountCalculatorStrongTest extends \PHPUnit\Framework\TestCase
{
public function testFiveOrMoreLoyaltyYearsGetsTwentyPercentOff(): void
{
$calc = new DiscountCalculator();
$this->assertSame(80.0, $calc->calculate(100.0, 5)); // exact boundary: 5
}
public function testFourLoyaltyYearsGetsOnlyFivePercentOff(): void
{
$calc = new DiscountCalculator();
$this->assertSame(95.0, $calc->calculate(100.0, 4)); // exact boundary: 4
}
}One mutation testing cycle for a single mutant
A killed mutant is good news about your tests; a surviving mutant on meaningful logic is a concrete, fixable gap.
Original source
|
v
Apply ONE mutation operator to ONE location
(e.g. flip >= to > on line 12)
|
v
Mutant source (rest of file unchanged)
|
v
Re-run ONLY tests that cover line 12 (from baseline coverage data)
|
+--> a test FAILS ---> mutant KILLED (tests caught the change)
|
+--> all tests PASS ---> mutant SURVIVED (tests missed the change)
|
+--> no test covers this line at all ---> NOT COVERED (separate category)
Repeat for every mutable location Infection can generate -> aggregate into MSIWhy mutation testing usually targets specific high-value paths, not the whole codebase
Because mutation testing is computationally expensive (potentially thousands of mutants, each requiring a test run) and because not every line carries equal risk, teams commonly scope Infection to their highest-value directories first — payment processing, auth, pricing/billing logic — rather than the entire codebase from day one, the same gradual-adoption instinct as a PHPStan baseline. A payment calculation with a 55% mutation score is a much more urgent finding than a view-formatting helper with the same score, so scoping the tool's effort (and the CI time budget) toward business-critical logic first gets more real bug-catching value per minute of CI time spent.
// Team dashboard tracks: "Line coverage: 96%" as the sole test-quality metric, // and treats reaching 100% as the goal for every new PR.
// Team dashboard tracks BOTH: line coverage (a floor — code that isn't // executed by any test can't be verified at all) AND mutation score / // MSI (the actual signal for whether tests verify meaningful behavior), // with --min-msi enforced in CI alongside a coverage floor.
Chasing 100% line coverage as the headline metric rewards tests that merely execute code, which is exactly the pattern that produces a high-coverage, low-mutation-score codebase — coverage is a necessary floor (untested code can't be verified at all) but mutation score is what actually measures whether the tests would catch a real bug, and a team that only tracks the former has no visibility into the gap.
What does it mean when a mutant 'survives' in an Infection report? A. The mutated code caused a fatal error unrelated to the test suite B. Every test in the suite still passed against the mutated code, meaning no test would have caught that specific kind of bug in that exact location C. The mutation was applied to a line with no test coverage at all D. Infection judged the mutation equivalent to the original and skipped scoring it
B. Every test in the suite still passed against the mutated code, meaning no test would have caught that specific kind of bug in that exact location — A surviving mutant means the test suite ran against deliberately altered (buggy) code and every test still passed — concrete evidence that a real bug matching that mutation, in that exact spot, would ship undetected by the current tests.
Why can a test suite have 100% line coverage and still have a low mutation score? A. Line coverage tools are unreliable and often report incorrect numbers B. Line coverage only measures whether a line executed during testing, not whether any test asserted something meaningful about the outcome of that line — a test can execute code while checking almost nothing about it C. Mutation testing always produces lower percentages than coverage by definition, regardless of test quality D. 100% coverage is mathematically impossible to achieve alongside mutation testing
B. Line coverage only measures whether a line executed during testing, not whether any test asserted something meaningful about the outcome of that line — a test can execute code while checking almost nothing about it — Coverage instrumentation tracks execution, not verification — a test that calls a function and asserts only its return type (or nothing at all) executes every line yet would not notice if the computed value were wrong, which is exactly what a low mutation score against high coverage reveals.
What is the practical difference between chasing higher line coverage and chasing a higher mutation score? A. They are two names for the exact same measurement B. Higher coverage means writing tests that execute more code; a higher mutation score requires writing (or strengthening) assertions that would actually fail if that code's behavior were subtly wrong — coverage is a floor, mutation score measures verification quality C. Mutation score can only be improved by deleting tests D. Coverage measures test speed, mutation score measures test count
B. Higher coverage means writing tests that execute more code; a higher mutation score requires writing (or strengthening) assertions that would actually fail if that code's behavior were subtly wrong — coverage is a floor, mutation score measures verification quality — Coverage answers 'did a test run this line'; mutation score answers 'would a test have noticed if this line were subtly wrong.' Improving coverage means exercising more code paths; improving mutation score means tightening assertions on the paths already exercised so they'd actually catch a realistic bug.
A PasswordStrengthChecker::isStrong(string $password): bool method has 100% line coverage and a mutation score of 45%. Write (in prose) two specific test cases you'd add, aimed at killing likely-surviving boundary and logical mutants, without seeing the method's source. Hint: Think about what a length-boundary mutator (>= vs >) and a logical-operator mutator (&& vs ||, or a negated condition) would attack: test at the exact minimum length that should just barely pass, and at one character below it that should just barely fail, with an assertSame/assertTrue/assertFalse on the exact expected boolean rather than a loose assertion.
Try it yourself — there's no single right answer here; compare your result against this lesson's examples.