Mutation Testing
Overview
Mutation testing is an automated quality gate that runs on pull requests. It measures how effectively the test suite detects real bugs by introducing small, deliberate changes (mutations) into the source code and checking whether the existing tests catch them. If a test fails when a mutation is applied, the mutant is killed -- the tests are doing their job. If all tests still pass, the mutant has escaped -- there is a gap in test coverage.
Traditional code-coverage metrics tell you which lines are executed during testing, but they do not tell you whether the tests actually verify correct behavior. Mutation testing closes that gap by confirming that the tests can detect meaningful changes to the code.
Reading the PR Report
When mutation testing runs on a pull request, a summary report is posted to the GitHub Step Summary (visible by clicking the "Mutation Testing" job in the PR checks). The report contains the following metrics:
| Metric | Description |
|---|---|
| MSI (Mutation Score Indicator) | The percentage of mutants that were killed by the test suite. A higher MSI means stronger test coverage. |
| Killed | The number of mutants that caused at least one test to fail. These are successfully detected changes. |
| Escaped | The number of mutants that did not cause any test to fail. These represent gaps in test coverage. |
| Errors | The number of mutants that caused the code to produce a fatal error or crash. These count as detected (the mutation was caught). |
| Timed Out | The number of mutants that caused the test suite to hang or exceed the time limit. These also count as detected. |
| Skipped | The number of mutants that were not evaluated, typically because they were in uncovered code. |
The MSI is calculated as:
MSI = (Killed + Errors + Timed Out) / Total Mutants * 100
What Escaped Mutants Mean
An escaped mutant means a deliberate change to the source code went unnoticed by the entire test suite. Common mutation types include:
| Mutation | Example |
|---|---|
| Arithmetic operator swap | + changed to - |
| Comparison boundary shift | > changed to >= |
| Boolean negation | true changed to false |
| Return value change | return $value changed to return null |
| Method call removal | A method call removed entirely |
When a mutant escapes, it means the test suite does not assert on the behavior that was changed. The code could break in exactly that way and no test would catch it.
What Action to Take
When the mutation report shows escaped mutants:
- Review the escaped mutants list -- the report identifies the file, line number, and type of mutation for each escape.
- Write or update tests -- add assertions that would fail if the mutation were applied. Focus on testing the specific behavior that the mutation altered, not just that the code runs without errors.
- Re-run the pipeline -- push the updated tests and verify the previously escaped mutants are now killed.
Not every escaped mutant requires immediate action. Some mutations affect code paths that are intentionally untested (e.g., logging, debug output). Use judgment when prioritizing which escapes to address, but treat a declining MSI as a signal that test quality needs attention.