Skip to main content

qa-test-mutate Reference

Extracted reference material for the qa-test-mutate skill. Contains multi-stack configs, scoring rubrics, and CI patterns.

Multi-Stack Framework Table​

StackFrameworkKey CI FeatureConfig FileInstall
JS/TSStrykerJS--incremental, coverageAnalysis: "perTest"stryker.config.mjsnpx --no-install stryker --version
PHPInfectionMSI/MCC/CCMSI metrics, PHPUnit/Pest compatibleinfection.json5vendor/bin/infection --version
Gogo-mutestingAST-based mutation, Avito fork recommendedCLI flagsgo-mutesting --version
Pythonmutmutmutmut results CLI output, mutate_only_covered_linessetup.cfg or CLImutmut --version

Known tool field values (non-normative, parsers MUST NOT validate against this list): "stryker", "infection", "go-mutesting", "mutmut"

New mutation tools may be added without schema changes. The tool field is free-form string type.

Mutation Run Commands​

JS/TS (StrykerJS)​

# Check availability (never bare npx)
npx --no-install stryker --version

# Full run
npx stryker run

# Incremental (PR scope -- changed files only)
npx stryker run --incremental

# With specific config
npx stryker run --configFile stryker.config.mjs

Config template (stryker.config.mjs):

/** @type {import('@stryker-mutator/api/core').PartialStrykerOptions} */
export default {
  mutate: ['src/**/*.ts', '!src/**/*.test.ts', '!src/**/*.spec.ts'],
  testRunner: 'vitest',
  reporters: ['html', 'json', 'progress'],
  coverageAnalysis: 'perTest',
  incremental: true,
  thresholds: {
    high: 80,
    low: 60,
    break: 60,
  },
};

Score source: reports/mutation/mutation.json -> mutationScore field

PHP (Infection)​

# Check availability
vendor/bin/infection --version

# Full run
vendor/bin/infection --min-msi=60 --min-covered-msi=80 --threads=4

# Incremental (PR scope -- changed lines only, 3.3x speedup)
vendor/bin/infection --threads=max \
  --git-diff-filter=AM --git-diff-base=main \
  --only-covering-test-cases --min-msi=70

# With specific config
vendor/bin/infection --configuration=infection.json5

# Show log
vendor/bin/infection --logger-json=infection-log.json

Incremental flags:

  • --git-diff-filter=AM: Only Added/Modified files
  • --git-diff-base=main: Compare against main branch
  • --git-diff-lines: Mutate only changed lines (not whole files) -- even more granular
  • --only-covering-test-cases: Run only tests that cover each mutant line (major speedup)

Config template (infection.json5):

{
  "$schema": "vendor/infection/infection/resources/schema.json",
  "source": {
    "directories": ["src"],
    "excludes": ["Tests", "Migrations"]
  },
  "phpUnit": {
    "configDir": "."
  },
  "mutators": {
    "@default": true
  },
  "minMsi": 60,
  "minCoveredMsi": 80
}

Score source: infection-log.json -> stats.msi field (Mutation Score Indicator)

Go (go-mutesting)​

# Check availability
go-mutesting --version

# Full run (Avito fork recommended)
go-mutesting ./...

# Specific packages
go-mutesting ./pkg/auth/... ./pkg/payment/...

Score source: parse stdout killed/total ratio (no native JSON output). Example output: The mutation score is 0.7250 (145 killed out of 200 total)

NEVER conflate with go test -cover output -- that measures code coverage, not mutation score.

Python (mutmut)​

# Check availability
mutmut --version

# Full run
mutmut run

# Run only covered lines (faster)
mutmut run --paths-to-mutate=src/

# Get results (parse this, NOT .mutmut-cache SQLite)
mutmut results

Score source: mutmut results CLI output -> parse survived/killed/total counts. Example output:

To apply a mutant on disk:
  mutmut apply <id>

Survived: 42
Killed: 158
Total: 200

NEVER parse .mutmut-cache -- this is an unstable internal SQLite database format.

Scoring Rubric and Thresholds​

Default Thresholds (configurable in jaan-to/config/settings.yaml)​

qa_mutation:
  thresholds:
    break_ci: 60        # Minimum score to pass CI
    target_new_code: 80  # Target for new/changed code
    critical_paths: 90   # Target for payments, auth, security
  feedback_loop:
    max_iterations: 3
    min_delta: 5         # Stop if improvement < 5 points

Score Interpretation​

Score RangeQuality LevelRecommendation
90-100%ExcellentTest suite is highly effective
80-89%GoodMeets target for new code
60-79%AdequatePasses CI, but improvement recommended
40-59%PoorBelow CI threshold, needs immediate attention
0-39%CriticalTest suite provides minimal value
nullUnknownMutation tool not available for this stack

Null Score Handling​

  • mutation_score: null means "not measured" (tool unavailable)
  • mutation_score: 0 means "measured zero" (all mutants survived)
  • Downstream consumers (qa-quality-gate, qa-test-run) must handle null by excluding from weighting
  • JSON parsers: treat null as absent signal, 0 as measured zero signal

CI Integration Patterns​

Incremental (PRs -- changed files only)​

Run mutation testing only on files changed in the PR for fast feedback:

# GitHub Actions example
- name: Mutation Testing (Incremental)
  if: github.event_name == 'pull_request'
  run: npx stryker run --incremental --mutate "$(git diff --name-only origin/main...HEAD | grep -E '\.(ts|js)$' | tr '\n' ',')"

Full (Nightly)​

Run full mutation testing suite on schedule:

# GitHub Actions example
on:
  schedule:
    - cron: '0 2 * * *'  # Nightly at 2 AM UTC

- name: Mutation Testing (Full)
  run: npx stryker run

Multi-Stack CI Matrix​

strategy:
  matrix:
    include:
      - stack: node
        cmd: npx stryker run
        cmd_incremental: npx stryker run --incremental
        score_file: reports/mutation/mutation.json
        concurrency: "concurrency: N in stryker.config.mjs"
      - stack: php
        cmd: vendor/bin/infection --logger-json=infection-log.json --threads=max
        cmd_incremental: vendor/bin/infection --logger-json=infection-log.json --threads=max --git-diff-filter=AM --git-diff-base=main --only-covering-test-cases
        score_file: infection-log.json
        concurrency: "--threads=max"
      - stack: go
        cmd: go-mutesting ./...
        cmd_incremental: go-mutesting ./...  # No incremental support
        score_file: stdout
        concurrency: "N/A (sequential)"
      - stack: python
        cmd: mutmut run && mutmut results
        cmd_incremental: mutmut run && mutmut results  # No incremental support
        score_file: stdout
        concurrency: "N/A (sequential)"

Performance Optimization Summary​

StackIncrementalConcurrencyPer-Test TargetingExpected Speedup
JS/TS (StrykerJS)--incremental (94% reuse)concurrency: NcoverageAnalysis: 'perTest'5-10x on re-runs
PHP (Infection)--git-diff-lines--threads=max--only-covering-test-cases3.3x
GoNoneNoneNoneN/A
Python (mutmut)SQLite cache (automatic)None (community fork)None~1.5x on re-runs

Survivors JSON Schema (Handoff Contract v1.0)​

{
  "schema_version": "1.0",
  "tool": "stryker",
  "run_timestamp": "2024-01-15T14:30:00Z",
  "mutation_score": 72.5,
  "total_mutants": 200,
  "killed": 145,
  "survived": 55,
  "survivors": [
    {
      "id": "mutant-001",
      "file": "src/services/auth.ts",
      "line": 42,
      "original": "return balance > 0;",
      "mutated": "return balance >= 0;",
      "mutator": "ConditionalBoundary",
      "status": "Survived"
    }
  ]
}

Field Requirements​

FieldTypeRequiredNotes
schema_versionstringYesAlways "1.0"
toolstringYesFree-form, not enum
run_timestampstring (ISO-8601)YesUTC timestamp
mutation_scorenumber or nullYesPercentage (0-100) or null if unavailable
total_mutantsintegerYesTotal mutants generated
killedintegerYesMutants killed by tests
survivedintegerYesMutants that survived
survivors[]arrayYesMay be empty
survivors[].idstringYesUnique within this run
survivors[].filestringYesRelative path to source
survivors[].lineintegerYesLine number in source
survivors[].originalstringYesOriginal code snippet
survivors[].mutatedstringYesMutated code snippet
survivors[].mutatorstringYesMutator type name
survivors[].statusstringYesAlways "Survived"