gjalla-test-audit

# Test Audit

Deep review of test suite quality to find tests that give false confidence, encode bugs, duplicate coverage, or are so heavily mocked they can't catch real regressions.

## Process

At a high level, you'll follow the steps below, then cross-reference load-bearing code with test imports. Importance can be ranked by production blast radius (which code is most depended-on — if you use gjalla, impact and change history surface this), recent bug history, and how deterministic the failure mode is.

### Phase 1: Orient

Understand the project's test infrastructure before diving in.

1. **Map test structure**: Find all test directories, count files per directory, identify naming conventions (`.test.ts`, `.pglite.test.ts`, `.integration.test.ts`, etc.)
2. **Identify test layers**: Which tests use real databases (PGlite, SQLite)? Which mock the ORM? Which mock at the service boundary? Which use `@vitest-environment node` vs `jsdom`?
3. **Identify critical components**: What are the security boundaries, data access layers, and core business logic? These are where false confidence is most dangerous.

### Phase 2: Hunt for anti-patterns

Launch parallel investigations across test layers. For each test file, read BOTH the test AND the source code it claims to test. The anti-patterns to find:

#### Anti-pattern 1: Reimplemented logic tests
Tests that never import the real code. Instead they redefine the logic inline and test their own copy. Signals:
- `@vitest-environment node` with no component/hook imports
- Local functions named `simulate*` or `handle*` that mirror source code
- Test file has zero imports from `src/` or source directories

These tests will NEVER catch a regression because they don't exercise the real code.

#### Anti-pattern 2: Tautological mock tests
Tests that mock the entire database/ORM chain with hardcoded returns, then assert those same hardcoded values. Signals:
- `mockReturnValue` / `mockResolvedValue` on `db.select().from().where()` chains
- Queue-based mock infrastructure (`_setSelectQueue`, `pushSelectResult`)
- Assertions like `expect(result).toEqual(mockReturnValue)` where `mockReturnValue` is what the mock was set up to return
- Builder pattern mocks where `.from()`, `.where()`, `.innerJoin()` all ignore their arguments

**Key test**: Could a bug in the real code (wrong table, wrong column, wrong WHERE clause, wrong JOIN) cause this test to fail? If no, the test is tautological.

#### Anti-pattern 3: Tests encoding wrong behavior
Tests whose assertions verify incorrect behavior that happens to match buggy source code. Signals:
- Test fixtures using field names that don't match the source (e.g., test uses `assignedTier` but source reads `subscriptionTier`)
- Inconsistent thresholds between services tested independently
- Mock return values that paper over logic the test claims to verify
- Test names that say one thing but assert another (e.g., "returns 403" but asserts `toBe(404)`)

**Key test**: Does the test's mock data match what real upstream code actually produces? Or was it hand-crafted to match the (possibly buggy) function under test?

#### Anti-pattern 4: Redundant companion tests
Tests that are fully covered by a more rigorous companion file. Signals:
- A `.test.ts` file that mocks the DB alongside a `.pglite.test.ts` file that tests real SQL for the same class
- An "integration" test that mocks at the same level as the "unit" test
- Multiple test files for the same source file with overlapping `describe`/`it` blocks

#### Anti-pattern 5: Placeholder tests
- `expect(true).toBe(true)`
- Tests with descriptive names but no real assertions
- Tests that call a mock and then assert the mock was called (tautology)

### Phase 3: Classify findings

Organize findings into tiers:

| Tier | Description | Action |
|------|-------------|--------|
| **Tier 1** | Tests that exercise zero real code (reimplemented logic, inline mock handlers) | Delete entire file |
| **Tier 2** | Files with mixed useful and tautological tests | Delete tautological sections, keep logic tests |
| **Tier 3** | Tautological tests that have a real companion (PGlite, integration) | Delete redundant mocked version |
| **Tier 4** | Tautological tests with NO real companion | Flag as dangerous false confidence. These need real tests written. |
| **Bugs** | Tests that encode wrong behavior in source code | Fix source code AND test |

### Phase 4: Report

Present findings as a structured report with:

1. **Tests encoding bugs** (highest priority) - these are masking real production issues
2. **Tier 1-3 deletions** with file paths and line counts
3. **Tier 4 gaps** - areas where coverage will honestly drop and needs real tests
4. **Impact analysis**: what gets more robust, what bugs may surface, coverage impact

### Phase 5: Clean up (if approved)

Execute in waves, running tests between each:

1. Delete Tier 1 files (zero real code tested)
2. Remove Tier 2 tautological sections
3. Remove Tier 3 redundant tests (where companion exists)
4. Fix tests encoding wrong behavior + fix source bugs
5. Run coverage to identify honest gaps
6. Write real tests for Tier 4 gaps (PGlite for DB-dependent code)

## Principles

- **A test that can't fail is worse than no test.** It inflates confidence without providing protection.
- **Mock at boundaries, not internals.** Mock external services (Supabase, Stripe, GitHub API). Don't mock your own database layer when you can use PGlite.
- **Round-trip tests catch key mismatches.** When function A writes data that function B reads, test them together using A's actual output as B's input. Hand-crafted fixtures can silently encode bugs.
- **Coverage numbers lie when tests are tautological.** A line is not "covered" if the mock intercepted every meaningful operation. Honest coverage comes from tests that exercise real code paths.
- **Delete before you write.** Remove false confidence first so the coverage report becomes an honest map of where you're strong and wher