refd

# refd

[![CI](https://github.com/emaitchess/refd/actions/workflows/ci.yml/badge.svg)](https://github.com/emaitchess/refd/actions/workflows/ci.yml) [![License: MIT](https://img.shields.io/github/license/emaitchess/refd)](LICENSE) [![Try refd.ai](https://img.shields.io/badge/try-refd.ai-111111)](https://refd.ai) [![Cloudflare Workers](https://img.shields.io/badge/Cloudflare-Workers-F38020?logo=cloudflare&logoColor=white)](https://workers.cloudflare.com/)

Open-source AI search monitoring — track how AI answers talk about any brand: visibility, mentions, citations, and rank across ChatGPT, Perplexity, Gemini, Google AI Mode, and Google AI Overviews. Use the hosted app at [refd.ai](https://refd.ai) or self-host the whole stack.

A Bun-workspace monorepo of three independently deployed Cloudflare Workers: **`apps/api`** (`api.refd.ai`) — the Hono API, OAuth, remote MCP, daily cron, and queue consumer, holding every binding; **`apps/dashboard`** (`dash.refd.ai`) — the React SPA as an assets-only Worker; **`apps/web`** (`refd.ai`) — the static Astro public site. Shared runtime-neutral code lives in **`packages/core`**. Data via BrightData (dataset scrapers + SERP API), stored in D1 (Drizzle) with gzipped raw payloads in R2.

refd also exposes the same workspace intelligence to AI agents through a remote
MCP connector. Its default `data:read` scope is read-only; the optional
`data:write` scope adds the bounded setup lifecycle, can start one onboarding
report, and can revoke the connection with explicit confirmation. The hosted
guide lives at [refd.ai/agents](https://refd.ai/agents); the
[MCP connector guide](docs/mcp.md) covers Claude, Claude Code, ChatGPT, generic
clients, personal access tokens for headless agents, and self-hosting setup.

One click from the guide installs it in [Cursor](cursor://anysphere.cursor-deeplink/mcp/install?name=refd&config=eyJ0eXBlIjoiaHR0cCIsInVybCI6Imh0dHBzOi8vYXBpLnJlZmQuYWkvbWNwIn0=) or [VS Code](vscode:mcp/install?name=refd&config=%7B%22type%22%3A%22http%22%2C%22url%22%3A%22https%3A%2F%2Fapi.refd.ai%2Fmcp%22%7D) — or `claude mcp add-json refd '{"type":"http","url":"https://api.refd.ai/mcp"}'`. Headless agents use a personal access token from **Settings → Personal access tokens** instead of OAuth.

## How it works

Accounts hold **workspaces**; each workspace tracks one brand — its competitors, its prompt set, its runs. A daily cron creates an idempotent run for every workspace eligible for scheduled monitoring, then fans out queue messages: one batch-snapshot trigger per dataset surface × sample (trigger → notify → fetch, with a backstop poll), plus one sync SERP call per prompt × sample for AI Overviews. Every answer is scored for every tracked entity (mentioned / cited / first-mention position), and the raw payload is archived gzipped in R2. A missing AI Overview is recorded as a valid "no AIO shown", not a failure. Run dispatch persists immutable launch plans and resumes with bounded backoff. Storage converges under queue redelivery, but BrightData has no idempotency key, so a crash after provider acceptance can repeat paid work.

## Develop

```bash
bun install
bun run dev    # applies local migrations, then runs all three Workers behind
               # Caddy on same-site subdomains over real HTTPS
```

Register on the login screen (business email + password ≥ 8 chars) — your first workspace is created automatically and drops you into a resumable setup wizard: name your brand, let it draft your description from your site, pick competitors and prompts, choose engines, then watch the first report fill in live. Every AI step falls back to typing it yourself. Standard accounts can create up to five workspaces, keep up to 25 active prompts in each, and enable up to three AI surfaces. Emails in `ADMIN_EMAILS` have no workspace or active-prompt cap and can enable all available surfaces.

`bun run dev` starts Caddy (`Caddyfile`) fronting all three Workers on same-site subdomains — Astro at **https://refdlocal.io**, the dashboard at **https://dash.refdlocal.io**, the API at **https://api.refdlocal.io** (add all three to `/etc/hosts` as `127.0.0.1`, and run `caddy trust` once). Real HTTPS on same-site subdomains exercises secure cross-origin cookies, CORS, OAuth redirects, and WebSocket streaming just like production. Caddy stops when dev exits.

Local secrets go in `apps/api/.dev.vars` (gitignored): `JWT_SECRET`, `BRIGHTDATA_API_TOKEN`, `BRIGHTDATA_WEBHOOK_SECRET` (optional locally; requires a publicly reachable `PUBLIC_BASE_URL`), `ADMIN_EMAILS` (comma-separated administrator and operator allowlist), and `EXA_API_KEY` (onboarding's competitor search, optional). Non-secret build-time origins (`VITE_API_ORIGIN`, `PUBLIC_DASHBOARD_ORIGIN`, …) are committed per environment in each app's `.env.development` / `.env.production`. Analytics is a self-hosted [Umami](https://umami.is) instance shared by the public site and the dashboard, so the activation funnel can span both: the web app needs `PUBLIC_UMAMI_HOST_URL`, `PUBLIC_UMAMI_WEBSITE_ID`, and `PUBLIC_ANALYTICS_HOSTNAME`, and the dashboard needs the same three as `VITE_*` (with the **same** website id). Leave any of them unset and the build emits no tracker at all, which is what a self-hosted deployment wants. Onboarding also uses two **bindings, not secrets** — Workers AI (`AI`) and Browser Rendering (`BROWSER`) — both `remote: true` in `apps/api/wrangler.jsonc`, so local dev proxies to the real services and the account needs both enabled.

- `bun run check` — typecheck (all workspaces) · `bun run lint` / `lint:fix` — Biome · `bun test` — unit tests
- `bun run build` — build all three Workers · `bun run deploy` — build, migrate D1, then deploy all three (the self-hosted/manual path; on refd.ai, merging to `main` deploys all three Workers automatically via Cloudflare)

Git worktrees are supported out of the box with [`wt`](https://github.com/emaitchess/wt): `wt create <name>` runs the checked-in `.wt/setup.sh`, which installs dependencies and copies local secrets and wrangler state, so a fresh worktree behaves like your main checkout.

## Self-host

```bash
wrangler queues create refd-ingest && wrangler queues create refd-ingest-dlq
wrangler secret put JWT_SECRET
wrangler secret put BRIGHTDATA_API_TOKEN
wrangler secret put ADMIN_EMAILS       # comma-separated admin/operator allowlist
wrangler secret put EXA_API_KEY        # optional: onboarding competitor search
bun run db:migrate:remote
bun run deploy
```

`bun run deploy` builds all three Workers and deploys them (`refd-api`, `refd-dashboard`, `refd-web`); point their custom domains at `api.`, `dash.`, and the apex. The API workspace's own `deploy` script runs remote D1 migrations first, so deploying `@refd/api` alone (CI or a Cloudflare dash deploy command) always brings the schema along. (The hosted deployment wires these deploy scripts to push-to-main via Cloudflare's Git integration, so refd.ai itself has no manual deploy step.) You'll need a Cloudflare account (Workers paid plan for Queues, plus Workers AI and Browser Rendering enabled for onboarding), a D1 database + R2 bucket (ids/names in `apps/api/wrangler.jsonc`), and a BrightData account: fill the dataset IDs in `apps/api/wrangler.jsonc` `vars` (dashboard → Web Scrapers → each AI scraper) and create a SERP API zone matching `BRIGHTDATA_SERP_ZONE`. Set the deployment origins (`PUBLIC_BASE_URL`, `DASHBOARD_ORIGIN`, `PUBLIC_SITE_ORIGIN`, `API_ORIGIN`) and `SCHEDULED_MONITORING_POLICY=all` for a self-hosted deployment; the checked-in `entitled` policy is for refd.ai and limits cron to active `pilot` or `subscribed` workspaces. Cron schedule (daily 06:00 UTC), samples, and geo also live in `apps/api/wrangler.jsonc`.

## Notes

- AI answers are non-deterministic: hosted scheduled runs use `SAMPLES=1` per prompt and surface; read trends across completed runs, not one sample. Operators can request additional samples for focused manual runs.
- Each standard full run is capped at 25 active prompts × 3 enabled surfaces × samples (25 × 3 × 1 = 75 records at the default). Quota scales with the number of workspaces eligible for scheduled monitoring.
- The public site is built to be read by agents as well as people: every page is also served as markdown at the same path with `.md` appended, `/llms.txt` follows the [llmstxt.org](https://llmstxt.org/) format and links those markdown versions, and `/llms-full.txt` inlines them all. Self-hosted deployments get this for free.
- Design system: `DESIGN.md`. Scoring/metrics contract: `docs/METRICS.md`. Remote MCP + OAuth: `docs/mcp.md`.

## Contributing

Contributions welcome. See [`CONTRIBUTING.md`](CONTRIBUTING.md) for setup and the
checks your PR must pass, and [`CODE_OF_CONDUCT.md`](CODE_OF_CONDUCT.md) for
community expectations. Found a security issue? Follow [`SECURITY.md`](SECURITY.md)
instead of opening a public issue.

## License

[MIT](LICENSE) © refd