ln-23-test-suite-auditor

# Test Suite Auditor

**Goal:** Audit the test portfolio as a read-only lifecycle and confidence system. Determine which important failures it detects, which evidence is untrustworthy or obsolete, and which additions, changes, consolidations, retirements, or explicit omissions produce the smallest sustainable portfolio.

**Execution contract:** Treat the ordered checkbox workflow below as this skill's Definition of Done. Track every checkbox as `PENDING`, then resolve it to `PROVEN` with concrete evidence, `CLEARED` with evidence that its conditional trigger is absent, or `UNPROVEN`; reading, mentioning, delegating, skipping, or tool failure is not proof.
Before returning, resolve every `PENDING`, count only `PROVEN` and `CLEARED` items as complete, apply this skill's verdict, decision, and approval rules to every `UNPROVEN`, and prepend **Checklist: X/Y complete**<br>**Incomplete: None | section/item — reason; outcome impact; exact next action**; list every `UNPROVEN` item.

## Tool Routing

| Need | Preferred tool | Use it when | Fallback |
|---|---|---|---|
| Source and test inventory | Native file listing, search, manifests, and test configuration | Mapping domains, test types, runners, fixtures, and generated areas | Repository tree plus known test entrypoints |
| Test-to-code relationships | Language server or host-native code intelligence | Mapping units, callers, implementations, routes, and test targets | Naming and path search verified by direct reads |
| Execution and trust | Repository-defined test commands through the shell | Establishing pass/fail state, timing, order dependence, or reproducibility | Inspect CI results and configuration; mark execution unavailable |
| Coverage and missed behavior | Existing coverage tools and reports | Coverage data is configured and comparable to source scope | Static behavior-to-test mapping; never invent percentages |
| Flake and isolation evidence | Repeated, shuffled, parallel, or seed-controlled runs supported by the repository | A test is suspected of order, time, randomness, or shared-state dependence | History, CI logs, and code-path evidence |
| Assertion strength | Test reads, failure output, and configured mutation testing | Determining whether tests fail for meaningful behavioral defects | Counterfactual reasoning tied to specific assertions |
| Framework semantics | Official test-runner or framework documentation | A finding depends on lifecycle, fixtures, retries, isolation, or mocking behavior | Primary-source web research; otherwise mark `UNVERIFIED` |

Run only safe test and diagnostic commands. Do not rewrite snapshots, update golden files, regenerate fixtures, or accept changed output during the audit.

## Evidence Rules

- Coverage indicates execution, not proof. Require an assertion or observable oracle for important behavior.
- A slow test is not low-value when it uniquely protects a critical journey; a fast test is not high-value when it proves framework behavior.
- A flaky failure must be separated from an intermittently failing product dependency or genuinely nondeterministic requirement.
- Deletion recommendations require proof that the test basis is obsolete or that other evidence covers every still-required behavior and failure mode with equal or better trust.
- Merge recommendations require demonstrated duplicate or fragmented coverage and must preserve distinct business and failure scenarios, oracle strength, and failure localization; a larger test is not inherently better.
- Known regression guards and the only proof of a rare critical edge case are not deletion candidates merely because a numeric value heuristic is low.
- A real dependency is not inherently a test defect. Judge whether its version, state, ownership, reset, availability, and failure evidence make the result reproducible.
- Keep portfolio action (`KEEP`, `ADD`, `UPDATE`, `MERGE`, `DELETE`, `NO_TEST`) separate from execution status (`PASS`, `FAIL`, `BLOCKED`, `UNPROVEN`, `QUARANTINED`). `NO_TEST` requires existing proof, another control, or explicit residual-risk acceptance.
- Do not require a new registry by default. Prefer traceability derivable from repository-native test paths, behavioral names, tags, requirements, CI configuration, and review evidence.
- External testing guidance becomes actionable only when it explains a concrete weakness in this suite.

## Checklist

### 1. Map the Portfolio and Baseline

- [ ] Detect test runners, configurations, commands, directories, fixtures, factories, snapshots, golden files, manual scripts, coverage, and mutation tooling.
- [ ] Map source domains and critical entrypoints to unit, integration, contract, end-to-end, and manual test surfaces.
- [ ] Trace requirements, product risks, incidents, public contracts, and changed behavior to tests and results where evidence exists; identify tests with no current test basis and basis elements with no credible proof.
- [ ] Read repository instructions and CI configuration to identify required suites, environment assumptions, retries, sharding, and exclusions.
- [ ] Run the smallest representative suites, then required test gates where feasible; record environment, duration, exit status, failures, skips, and retries.
- [ ] Separate generated, vendored, example, migration-history, and infrastructure fixtures from product tests before evaluating the portfolio.
- [ ] Keep the audit read-only and disclose any caches or test artifacts created by permitted commands.

### 2. Audit Product Value and Coverage

- [ ] Identify uniquely critical local logic: money, authentication, authorization, data integrity, algorithms, domain rules, destructive operations, and irreversible workflows.
- [ ] Trace each critical behavior to at least one test whose oracle would fail for the corresponding defect; name/path matches and line coverage are only discovery evidence.
- [ ] Identify tests that merely re-prove language, framework, database engine, ORM, HTTP client, cryptography, s