caliber-testing

# Caliber Testing

## Critical

- All test files MUST be placed in `__tests__/` directories parallel to source files: `src/[module]/__tests__/[module].test.ts`
- Register any new `__tests__/` directories in `vitest.config.ts`'s `include` glob (already configured: `src/**/*.test.ts`)
- NEVER mock `src/test/setup.ts` — it is the global LLM provider mock already applied to all tests
- Environment variable tests MUST save `process.env` in `beforeEach`, restore in `afterEach`, and explicitly delete env vars to test absence
- Temporary file/directory cleanup MUST happen in `afterEach`, not in individual test cleanup. Use `fs.rmSync(dir, { recursive: true, force: true })`
- When a test requires unmocking modules mocked in global setup, call `vi.unmock('../module.js')` BEFORE the import statement
- Run `pnpm test` locally and `pnpm test:coverage` before committing to verify coverage thresholds (lines: 50, functions: 50, branches: 50, statements: 50)

## Instructions

### Step 1: Create the test file in the correct directory
Create `src/[module]/__tests__/[module].test.ts`. The parent source file is `src/[module]/[module].ts`.

**Verify**: The `__tests__` directory exists at the same level as the source file being tested.

### Step 2: Import test framework
At the top of every test file, import from `vitest`:
```typescript
import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest';
```

Add additional imports based on what you're testing:
- File system: `import fs from 'fs'; import path from 'path'; import os from 'os';`
- Temporary files: Use `fs.mkdtempSync(path.join(os.tmpdir(), 'caliber-prefix-'))`
- Exec: `import { execSync } from 'child_process';`

**Verify**: All required test utilities are imported before test definitions.

### Step 3: Set up module mocking (if needed)
If testing a module that imports other modules you want to mock:

```typescript
vi.mock('../config.js', () => ({
  loadConfig: vi.fn(),
  writeConfigFile: vi.fn(),
}));
```

For complex mocks with test-time factory functions (hoisted):
```typescript
const { mockLoadConfig } = vi.hoisted(() => ({
  mockLoadConfig: vi.fn(),
}));

vi.mock('../config.js', () => ({
  loadConfig: () => mockLoadConfig(),
}));
```

For unmocking global setup mocks (e.g., to test llm/index.js itself):
```typescript
vi.unmock('../index.js');
```

Place all `vi.mock()` and `vi.unmock()` calls BEFORE importing the module under test.

**Verify**: Mock declarations appear before the import of the module being tested.

### Step 4: Organize tests in describe blocks
Group related tests with `describe()`:
```typescript
describe('functionName', () => {
  it('returns X when Y', () => {
    // test body
  });
});
```

**Verify**: Each `it()` test has a clear, complete assertion.

### Step 5: Manage environment and process state
For tests that modify `process.env` or `process.argv`:

```typescript
describe('config tests', () => {
  const originalEnv = process.env;
  const originalArgv = process.argv;

  beforeEach(() => {
    process.env = { ...originalEnv }; // Copy, not reference
    process.argv = [...originalArgv];
    delete process.env.SPECIFIC_VAR; // Explicitly remove vars to test absence
  });

  afterEach(() => {
    process.env = originalEnv;
    process.argv = originalArgv;
  });

  it('tests env var behavior', () => {
    process.env.MY_VAR = 'test';
    // test code
  });
});
```

**Verify**: `beforeEach` creates a copy of env/argv; `afterEach` restores originals; unused env vars are explicitly deleted with `delete process.env.VAR`.

### Step 6: Manage temporary files and directories
For file system tests, create temporary directories and clean them up:

```typescript
describe('file tree', () => {
  const dirs: string[] = [];
  
  afterEach(() => {
    for (const d of dirs) {
      try { fs.rmSync(d, { recursive: true, force: true }); } catch {}
    }
    dirs.length = 0;
  });

  it('processes files', () => {
    const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'caliber-test-'));
    dirs.push(tmp);
    // use tmp directory
  });
});
```

**Verify**: All temporary directories are pushed to a cleanup array and removed in `afterEach`.

### Step 7: Handle mock state cleanup
Before each test, clear mock call history to avoid pollution between tests:

```typescript
beforeEach(() => {
  vi.clearAllMocks(); // Reset all mock call counts and return values
});

afterEach(() => {
  vi.restoreAllMocks(); // Restore original implementations
  vi.resetModules(); // Reset cached module imports (if you reload modules)
});
```

**Verify**: `beforeEach` calls `vi.clearAllMocks()` for providers and `afterEach` calls `vi.restoreAllMocks()`.

### Step 8: Test assertions
Write assertions using `expect()`. Match the patterns from existing tests:

```typescript
// Simple checks
expect(value).toBe(expected);
expect(array).toContain(item);
expect(fn).toThrow('error message');

// Instance checks
expect(obj).toBeInstanceOf(ClassName);

// Mock checks
expect(mockFn).toHaveBeenCalledTimes(1);
expect(mockFn).toHaveBeenCalledWith(arg);

// File system checks
expect(fs.existsSync(path)).toBe(true);
```

**Verify**: Each test has at least one assertion and uses appropriate `expect()` matchers.

### Step 9: Run tests
Run tests locally before committing:
```bash
pnpm test                # Run all tests in watch mode
pnpm test:coverage       # Check coverage thresholds
pnpm test -- src/my/path/__tests__/my.test.ts  # Run single test file
```

**Verify**: All tests pass and coverage thresholds are met (lines: 50%, functions: 50%, branches: 50%).

## Examples

### Example 1: Testing a module with environment variables (config.test.ts pattern)

**User asks**: "Write tests for my config loader that reads from env vars and a config file."

**Actions taken**:
1. Create `src/lib/__tests__/config.test.ts`
2. Import test framework and fs module
3. Mock `fs` and `os`
4. Save/restore `process.env` in beforeEach/afterEach
5. Delete specific env vars to test absence
6. Test each b