create-cli-e2e

# CLI E2E Test Development

Write, expand, and maintain end-to-end tests for CLI commands in `ts/e2e-tests/cli/`.

## When to Use

- Adding a new e2e test suite for a CLI command
- Modifying or extending an existing CLI e2e test
- Debugging a failing CLI e2e test
- Reviewing whether a CLI command's output contract is properly tested

For CLI **design** (arguments, flags, help text, UX), see the `create-cli` skill.
For CLI **implementation** (Effect patterns, services, command registration), see the `implement-cli-command` skill.

## Architecture

Each CLI e2e test runs the compiled `composio` binary inside a **scratch Debian Docker container**. The binary is self-contained (built via `bun build --compile`) — no Node, Bun, or pnpm exists in the runtime image.

Key properties:

- Each test suite = a directory under `ts/e2e-tests/cli/<suite-name>/`
- Use `runCmd` only. Never use `runFixture` (throws an error for CLI tests). Never set `usesFixtures`.
- **Each `runCmd` call creates a fresh container.** No state persists between calls.
- Commands run inside `sh -c '...'` — POSIX shell only, no bash-isms.
- Containers have network access — API-calling commands work.
- `HOME=/tmp`, cache dir is `/tmp/.composio/` — auth passes via env vars only.
- `process.stdout.isTTY` is always `false` inside Docker — the CLI always runs in piped mode.

### What "piped mode" means for tests

Inside Docker, the composio binary's stdout is never a TTY. This triggers the CLI's piped-mode behavior (see `ts/packages/cli/AGENTS.md` § "Output Conventions"):
- `ui.output()` writes to stdout
- All Clack decoration is suppressed
- stderr is empty for successful commands

## File Structure

For a new test suite `<suite-name>`, create 2 files:

```
ts/e2e-tests/cli/<suite-name>/
├── e2e.test.ts     # Test file
└── package.json    # Package manifest
```

### Naming Conventions

- **Directory**: hyphen-separated lowercase matching the command structure
  - `version`, `whoami`, `toolkits-list`, `tools-info`, `auth-configs-list`, `connected-accounts-link`
- **Package name**: `@e2e-tests/cli-<suite-name>`
  - `@e2e-tests/cli-version`, `@e2e-tests/cli-toolkits-list`

### package.json Template

```json
{
  "name": "@e2e-tests/cli-<suite-name>",
  "version": "0.0.0",
  "private": true,
  "type": "module",
  "scripts": {
    "typecheck": "tsc --noEmit",
    "test:e2e": "bun test e2e.test.ts",
    "test:e2e:cli": "bun test e2e.test.ts"
  },
  "devDependencies": {
    "@e2e-tests/utils": "workspace:*"
  }
}
```

## Choosing a Test Pattern

Use this decision tree to select the right pattern:

```
Does the command need env vars (e.g., COMPOSIO_API_KEY)?
├─ No → Is the output deterministic (exact value known at test time)?
│       ├─ Yes → Pattern A (simple command, exact assertions)
│       └─ No  → Pattern D (fuzzy assertions: toContain, toBeGreaterThan)
└─ Yes → Is the output deterministic given the env?
         ├─ Yes → Pattern B (env vars + exact assertions)
         └─ No  → Pattern D (env vars + fuzzy assertions)

Are you testing an error case (missing args, invalid input)?
└─ Yes → Pattern C (non-zero exit code, stderr non-empty)

Does the command perform an action with no machine-readable output?
└─ Yes → Pattern E (exitCode=0, stdout empty, no redirect test)
```

## Test Patterns

### Pattern A: Simple Command, No Env Vars (Canonical)

For commands that produce deterministic output without needing authentication. This is the base pattern — all other patterns are variations of this.

**Reference**: `ts/e2e-tests/cli/version/e2e.test.ts`

```typescript
/**
 * CLI version command e2e test
 *
 * Verifies that the compiled composio CLI behaves correctly in a scratch container.
 */

import { e2e, sanitizeOutput, type E2ETestResult, type E2ETestResultWithFiles } from '@e2e-tests/utils';
import { TIMEOUTS } from '@e2e-tests/utils/const';
import { describe, it, expect, beforeAll } from 'bun:test';
import cliPkg from '../../../packages/cli/package.json' with { type: 'json' };

e2e(import.meta.url, {
  versions: {
    cli: ['current'],
  },
  defineTests: ({ runCmd }) => {
    const expectedVersion = String(cliPkg.version ?? '').trim();
    let versionResult: E2ETestResult;
    let redirectedResult: E2ETestResultWithFiles<'out.txt'>;

    beforeAll(async () => {
      versionResult = await runCmd('composio version');
      redirectedResult = await runCmd({
        command: 'composio version > out.txt',
        files: ['out.txt'],
      });
    }, TIMEOUTS.FIXTURE);

    describe('composio version', () => {
      it('exits successfully', () => {
        expect(versionResult.exitCode).toBe(0);
      });

      it('stdout matches snapshot', () => {
        expect(sanitizeOutput(versionResult.stdout)).toBe(expectedVersion);
      });

      it('stderr matches snapshot', () => {
        expect(versionResult.stderr).toBe('');
      });
    });

    describe('stdout redirection to out.txt', () => {
      it('exits successfully', () => {
        expect(redirectedResult.exitCode).toBe(0);
      });

      it('stdout is empty', () => {
        expect(redirectedResult.stdout).toBe('');
      });

      it('stderr is empty', () => {
        expect(redirectedResult.stderr).toBe('');
      });

      it('out.txt matches snapshot', () => {
        expect(sanitizeOutput(redirectedResult.files['out.txt'])).toBe(expectedVersion);
      });
    });
  },
});
```

**Structure summary:**
- Two test groups: **command execution** (stdout has data, stderr empty) + **stdout redirection** (file has data, Docker stdout/stderr both empty)
- Use `sanitizeOutput()` on stdout and file contents before assertions
- Use `TIMEOUTS.FIXTURE` for the `beforeAll` that runs Docker commands

### Pattern B: Command Requiring Env Vars

**Same as Pattern A**, with three additions:

**Reference**: `ts/e2e-tests/cli/whoami/e2e.test.ts`

1. **Type augmentation** for compile-time safety on `Bun.env`:
   ```typescript
   declare module 'bun' {
     interface Env {
       COMPOSIO_API_