five46

# five46

[![npm version](https://img.shields.io/npm/v/five46.svg)](https://www.npmjs.com/package/five46)
[![npm downloads](https://img.shields.io/npm/dm/five46.svg)](https://www.npmjs.com/package/five46)
[![license: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](./LICENSE)
[![node](https://img.shields.io/node/v/five46.svg)](https://www.npmjs.com/package/five46)

**An autonomous AI testing agent that verifies your app or API actually works while you're still building it — fully local, using your own LLM key.**

You just changed something, and you want to know — right now, against the real running thing — whether it actually works, without first writing a test yourself. Give five46 a plain-English goal — "log in and confirm the dashboard loads," "create a user via POST, then confirm it via GET" — and an LLM, using your own OpenAI, Anthropic, Gemini, Groq, or AWS Bedrock key, drives your real app or real API, one real action at a time, and tells you honestly whether it worked, with a root-cause hypothesis if it didn't. Once it does, that exact run is captured as a real, standalone Playwright (or `node:test`) spec you keep — so the same check that helped you while you were building the feature becomes a permanent regression test afterward, with no five46 or LLM involved in ever running it again.

![five46 driving a real browser through a login, add-to-cart, and checkout flow, then confirming the order was placed](./assets/demo.gif)

> **Status:** early proof of concept, verified end-to-end against real live LLM keys across dozens of real-world sites and APIs.

If five46 is useful to you, a ⭐ on [GitHub](https://github.com/sekharsdet/five46) helps other people find it — much appreciated!

## Why five46, and how it's different

Most testing tools assume you already have a suite to run. five46 is built for the moment *before* that — mid-feature, before a test exists at all. Point it at what you're building, describe the outcome you expect in plain English, and keep re-running it as you keep changing code; once it's solid, the run it just did becomes your regression test, not a separate thing you write afterward.

Most AI-driven test-generation tools also run in a cloud sandbox: your app's traffic, screenshots, and DOM leave your machine and go through a third-party service you don't control. five46 is the opposite bet — **everything runs on your laptop**, using a key you already pay for, and the *only* thing that ever leaves your machine is the text sent to your chosen LLM provider on each step (always disclosed, never hidden). If your organization can't adopt a cloud-hosted AI testing platform for compliance or trust reasons, this is built for exactly that constraint.

It's also not a black box: every run ends with a real `.spec.ts`/`.test.mjs` file you can read, diff, commit to your repo, and run in CI with plain `npx playwright test` — no vendor lock-in, no proprietary runner.

## Features

- **Bring your own key (BYOK)** — OpenAI, Anthropic, Gemini, Groq, or AWS
  Bedrock. Your key, your usage, your cost.
- **Fully local** — no cloud sandbox, no tunneling for local dev servers.
  Nothing but the LLM calls ever leaves your machine.
- **Browser and API testing** — drive a real Chromium browser, or drive
  real HTTP requests directly, from the same agentic engine.
- **Real, standalone output** — every successful run writes a plain
  Playwright `.spec.ts` (or `node:test` script for API tests) you can
  re-run any time, with no five46 or LLM involved.
- **Session reuse** — log in once, capture the session, reuse it across
  runs without paying the LLM cost of logging in every time.
- **Self-healing selectors** — a stale selector gets one bounded, disclosed
  recovery attempt instead of just failing the step.
- **Resilient generated specs** — when a real, live check confirms
  Playwright's own `getByRole()` resolves uniquely to the exact element a
  step acted on, the generated spec prefers it over a positional CSS
  selector, since it's far more resistant to future DOM changes. Falls back
  to the always-correct selector automatically wherever that check can't be
  made — never changes what the live run itself does.
- **Root-cause hypotheses** — a failed assertion gets an LLM-generated
  hypothesis for what likely went wrong and what to check next.
- **MCP server** — expose `five46_test`/`five46_api` as tools an
  IDE-embedded AI assistant (Claude Code, Cursor, etc.) can call directly.
- **Safe by default** — API testing is read-only unless you explicitly
  unlock writes/deletes; destructive-looking browser clicks are blocked by
  default too.
- **Flaky-test detection** — `--repeat N` runs the same goal N times and
  reports whether the outcome/behavior actually stayed the same.
- **Diffing** — `five46 diff` compares two generated run files directly.
- **Project management** — `five46.config.json` + `--project` for
  reusable, named target defaults (url, session, safety flags).
- **Video replay** — `--record-video` records the whole session as a
  `.webm`.
- **Structured planning** — on by default, one extra upfront LLM call plans
  the whole goal, then most steps execute directly against the real
  page/response with no further live decision needed; `--no-structured-plan`
  opts back into the fully-adaptive, live-decision-every-step loop.
- **Fast per-step decisions** (`--fast-steps`, opt-in) — on Groq/Gemini,
  swaps in a genuinely faster model tier for the high-frequency per-step
  decision only; the upfront plan always uses your configured model. No
  effect on OpenAI/Anthropic/Bedrock, already at their fastest reliable
  tier. Opt-in, not default — see "Fast per-step decisions" below.
- **Story mode** (`--story`) — splits a raw, multi-AC user story into
  independent goals and runs them with bounded concurrency, reporting a
  clear pass/fail per acceptance criterion. See "Story mode" below.

## five46 vs. cloud AI testing platforms

| | five46 | Typical cloud AI testing platform |
|---|---|---|
| Where it runs | Your machine, fully local | Their cloud sandbox |
| What leaves your machine | Only the text sent to your LLM provider per step (disclosed) | Your app's traffic, screenshots, DOM, credentials |
| Pricing model | BYOK — you pay your LLM provider directly, at cost | Usage-based platform subscription on top of their own LLM cost |
| Output | A real, standalone `.spec.ts`/`.test.mjs` file you own, re-runnable with plain Playwright/`node:test` | Usually tied to their own runner/dashboard |
| Best fit | Teams that can't send app data to a third party, or want to run tests entirely offline/on-prem | Teams that want a managed, zero-setup service and don't mind the tradeoff |

Not a knock on cloud platforms — it's a genuinely different tradeoff (their infra vs. your own key and your own machine), and the right choice depends on what your organization is allowed to send off-machine.

## Installation

```bash
npm install -g five46

npm install --save-dev playwright @playwright/test   # one-time, if your project doesn't already have it
npx playwright install chromium                       # one-time, downloads the browser
```

Or run it without installing globally:

```bash
npx five46 test http://localhost:3000 --goal "log in and confirm the dashboard loads"
```

<details>
<summary>Building from source instead (for contributing to five46 itself)</summary>

```bash
git clone https://github.com/sekharsdet/five46.git
cd five46
npm install
npm run build
node dist/cli.js test http://localhost:3000 --goal "..."
```

</details>

## Configuration

One-time setup (same shape as `gh auth login`/`aws configure`):

```bash
five46 config
```

This prompts for an LLM provider + key, masking secret input, and saves it
to `~/.five46/config.json` (user-only file permissions). Or set
environment variables instead — these always take priority over the saved
config, which is useful for CI:

```bash
export FIVE46_LLM_PROVIDER=openai   # or: anthropic, gemini, groq, bedrock
export FIVE46_LLM_API_KEY=sk-...    # for bedrock, use your AWS region instead
```

### Getting a key

Don't have a key yet? Pick whichever's easiest to get, or whichever you
already use — five46 calls one small, cheap model per provider on every
step (never a "flagship" model), so per-run cost is low regardless of
which one you pick. **If wall-clock speed is what you care about most,
pick Groq** — its whole differentiator is LPU-based inference hardware
built specifically for fast token generation, meaningfully faster
round-trips than typical GPU-hosted inference for an equivalent-size
model. Since a run's time is dominated by LLM round-trip latency (not
five46's own code), the provider you pick is the single biggest lever
you control over how fast a run feels:

| Provider | Get a key at | Notes |
|---|---|---|
| **Groq** | [console.groq.com/keys](https://console.groq.com/keys) → "Create API Key" | Free tier, no credit card required, generous rate limits — also the fastest provider here, built on inference-optimized hardware. |
| **Gemini** | [aistudio.google.com](https://aistudio.google.com/apikey) → "Get API key" | Free tier, no credit card required — the fastest path to a first successful run. |
| **OpenAI** | [platform.openai.com/api-keys](https://platform.openai.com/api-keys) → "Create new secret key" | Account creation is free, but a key can't make real calls until you add a payment method — no meaningful free tier. |
| **Anthropic** | [console.anthropic.com/settings/keys](https://console.anthropic.com/settings/keys) → "Create Key" | Same shape as OpenAI — you can browse the console for free, but need billing set up before a key actually works. |
| **AWS Bedrock** | No key — see below | Uses your existing AWS credentials instead of an API key. |

The model each provider calls: `gpt-4o-mini` (OpenAI), `claude-3-5-haiku-latest`
(Anthropic), `gemini-flash-latest` (Gemini), `llama-3.3-70b-versatile` (Groq),
`anthropic.claude-3-5-haiku-20241022-v1:0` (Bed