fitter

# Fitter — web data for AI agents

[![Release](https://img.shields.io/github/v/release/PxyUp/fitter)](https://github.com/PxyUp/fitter/releases)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://github.com/PxyUp/fitter/blob/master/LICENSE)
[![Go Reference](https://pkg.go.dev/badge/github.com/PxyUp/fitter.svg)](https://pkg.go.dev/github.com/PxyUp/fitter)
[![Sponsor](https://img.shields.io/badge/Sponsor-%E2%9D%A4-db61a2)](https://github.com/sponsors/PxyUp)

**Fitter turns any website or API into structured JSON — declaratively.** One JSON/YAML config describes where the data lives (HTTP request, headless browser, file, static value) and what to extract (JSON paths, CSS selectors, XPath). No code, no brittle scraping scripts.

Because configs are plain data, **LLMs can author them**. The built-in [MCP server](#how-to-use-fitter_mcp) lets Claude Code, Claude Desktop, or any MCP client write and run scraping pipelines on your machine, on demand:

> *"Get the top 5 HackerNews stories with titles and scores"* → the model authors a fitter config, validates it, runs it locally, and gets clean JSON back.

**One engine, five ways to use it:**

| | |
|---|---|
| 🤖 **Fitter MCP** | [MCP server](#how-to-use-fitter_mcp) exposing fitter to Claude Code, Claude Desktop and any MCP client |
| 🧠 **Fitter Agent** | AI-powered CLI: natural language → config → executed result |
| 🖥 **Fitter CLI** | run configs locally for test/debug/home usage |
| 📦 **Fitter Lib** | embed the engine in your own Go program |
| ⚙️ **Fitter** | long-running service mode with scheduling & notifications |

**Why fitter for AI agents?**

- **Declarative & auditable** — the agent produces a config you can read, save and re-run, not throwaway code
- **Local-first** — all fetching happens on your machine; no third-party scraping API, no keys, no per-request billing
- **Batteries included** — HTTP client, headless browser (Playwright/Chromium/Docker), JSON/HTML/XML/XPath parsing, pagination, cached references, host rate-limits — in a single static binary
- **Reusable** — what the agent authored today becomes tomorrow's cron job or service config

![](https://github.com/PxyUp/fitter/blob/master/demo.gif)


# How to use Fitter_MCP

Fitter MCP is a [Model Context Protocol](https://modelcontextprotocol.io) server (stdio transport) which lets any MCP client — Claude Code, Claude Desktop, IDE assistants, custom agents — run Fitter configs and get structured JSON back.

### Quick start (Claude Desktop — one click)

Download `fitter-mcp-<os>-<arch>.mcpb` from the [release page](https://github.com/PxyUp/fitter/releases) and open it — Claude Desktop installs the server automatically.

### Quick start (Claude Code)

```bash
# 1. get the binary: download fitter_mcp_<version>-<os>-<arch> from the release page
#    https://github.com/PxyUp/fitter/releases — or build it from source:
go build -o fitter_mcp ./cmd/mcp

# 2. register it once, available in every project
claude mcp add fitter -s user -- "$(pwd)/fitter_mcp"
```

Then just ask:

> *Get the top 5 HackerNews stories with titles and scores using fitter*

The model calls `fitter_config_reference`, authors a config, optionally checks it with `fitter_validate_config` and executes it via `fitter_run` — all data fetching happens locally on your machine. For a ready-made pipeline try [examples/config_morning_briefing.json](https://github.com/PxyUp/fitter/blob/master/examples/config_morning_briefing.json):

> *Run examples/config_morning_briefing.json with fitter and give me the briefing*

### Register in Claude Desktop

```json
{
  "mcpServers": {
    "fitter": {
      "command": "/path/to/fitter_mcp"
    }
  }
}
```

### Tools

| Tool | Description |
|------|-------------|
| `fitter_run` | Run a Fitter config passed inline (JSON or YAML string) and return the extracted data as JSON. Accepts an optional `input` value available in the config via `{{{FromInput=.}}}` / `{{{FromInput=json.path}}}` |
| `fitter_run_file` | Same as `fitter_run` but reads the config from a local `.json`/`.yaml` file |
| `fitter_run_url` | Same as `fitter_run` but downloads the config from an HTTP(S) URL, e.g. a raw GitHub link |
| `fitter_validate_config` | Validate a config without executing it (structure, `response_type`, connector data source, model). Useful while iterating on a config |
| `fitter_config_reference` | Return a condensed reference of the whole config format (connectors, parsers, model/field schema, placeholders, notifiers, references, limits) with working examples, so the model can author configs without external docs |

The reference is also exposed as MCP resource `fitter://config-reference` for clients that support resources.

The config format is exactly the same as for [Fitter_CLI](#how-to-use-fitter_cli): a top-level object with `item` (required), `limits` and `references`. [Notifiers](#notifiers) work too (the result is additionally pushed to http/telegram/redis/file/console); `trigger_config` and `http_server` are service-mode only and are ignored in MCP calls.

### Remote / hosted mode (streamable HTTP)

By default `fitter_mcp` talks stdio. Pass `--http` to serve the [streamable HTTP transport](https://modelcontextprotocol.io/docs/concepts/transports) instead — for a shared team server, a container, or any remote deployment:

```bash
# serve MCP at http://<host>:8080/mcp (health probe at /healthz)
FITTER_MCP_AUTH_TOKEN=my-secret fitter_mcp --http :8080

# register the remote endpoint in Claude Code
claude mcp add --transport http fitter http://localhost:8080/mcp --header "Authorization: Bearer my-secret"
```

- `--http <addr>` (env `FITTER_MCP_HTTP_ADDR`) — listen address; stdio mode when empty
- `FITTER_MCP_AUTH_TOKEN` — when set, every `/mcp` request must send `Authorization: Bearer <token>`; without it the endpoint is unauthenticated, so bind to localhost or put it behind a proxy
- `--stateless` (env `FITTER_MCP_STATELESS=true`) — no per-session state, so replicas can sit behind a load balancer without sticky sessions

The server shuts down gracefully on SIGINT/SIGTERM.

#### Docker

A slim multi-arch image (`linux/amd64` + `linux/arm64`) ships with every release:

```bash
# hosted HTTP mode
docker run --rm -p 8080:8080 \
  -e FITTER_MCP_HTTP_ADDR=:8080 \
  -e FITTER_MCP_AUTH_TOKEN=my-secret \
  ghcr.io/pxyup/fitter-mcp:latest

# or stdio mode, spawned by the MCP client
claude mcp add fitter -s user -- docker run --rm -i ghcr.io/pxyup/fitter-mcp:latest
```

The image contains only the fitter binary and CA certificates: server/static/file connectors work, browser connectors (`chromium`/`docker`/`playwright`) do not — use a release binary on a host with a browser for those.

### Environment variables
1. **FITTER_PLUGINS** - string[""] - [path for plugins folder](https://github.com/PxyUp/fitter/blob/master/examples/plugin/README.md), same as the `--plugins` flag of Fitter/Fitter_CLI
2. **FITTER_MCP_HTTP_ADDR** - string[""] - listen address for [remote mode](#remote--hosted-mode-streamable-http), same as `--http`
3. **FITTER_MCP_AUTH_TOKEN** - string[""] - bearer token protecting the HTTP endpoint
4. **FITTER_MCP_STATELESS** - bool[false] - stateless HTTP transport, same as `--stateless`

# Recipes

Complete, tested configs showing the main patterns. All of them run unchanged via [Fitter_MCP](#how-to-use-fitter_mcp) (`fitter_run_file`), [Fitter_CLI](#how-to-use-fitter_cli) or the [library](#use-like-a-library) — more in [examples/](https://github.com/PxyUp/fitter/tree/master/examples).

### Scrape a page that has no API, enrich from one that does

GitHub trending has no official API — scrape the HTML for repo slugs (`html_attribute` reads the `href`), then fan each one out into the GitHub REST API with `{PL}`:

[examples/config_github_trending.json](https://github.com/PxyUp/fitter/blob/master/examples/config_github_trending.json)

```json
{
  "item": {
    "connector_config": {
      "response_type": "HTML",
      "url": "https://github.com/trending",
      "server_config": { "method": "GET", "headers": { "User-Agent": "Mozilla/5.0 (fitter demo)" } }
    },
    "model": {
      "array_config": {
        "root_path": "article.Box-row h2 a",
        "length_limit": 5,
        "item_config": {
          "field": {
            "type": "string",
            "html_attribute": "href",
            "generated": { "model": {
              "type": "object",
              "connector_config": {
                "response_type": "json",
                "url": "https://api.github.com/repos{PL}",
                "server_config": { "method": "GET", "headers": { "User-Agent": "fitter-demo" } },
                "null_on_error": true
              },
              "model": { "object_config": { "fields": {
                "repo": { "base_field": { "type": "string", "path": "full_name" } },
                "stars": { "base_field": { "type": "int", "path": "stargazers_count" } },
                "language": { "base_field": { "type": "string", "path": "language" } }
              } } }
            } }
          }
        }
      }
    }
  },
  "limits": { "host_request_limiter": { "api.github.com": 2 } }
}
```

```json
[{"repo": "block/buzz", "stars": 6214, "language": "Rust"}, {"repo": "koala73/worldmonitor", "stars": 71179, "language": "TypeScript"}]
```

### Join on a JSON field with an expression

When array items are objects, the join key lives inside them — pull it out with `{{{FromExp=...}}}` ([expr-lang](https://expr-lang.org/) over `fRes`, the current item). Book search → author details, search query supplied via `input`:

[examples/config_book_authors.json](https://github.com/PxyUp/fitter/blob/master/examples/config_book_authors.json)

```json
"url": "https://openlibrary.org/authors/{{{FromExp=fromJSON(fRes).author_key[0]}}}.json"
```

```bash
./fitter_cli --path=examples/config_book_authors.json --input=dune
```

```json
[{"title": "Dune", "year": 1965, "author": {"name": "Frank Herbert", "born": "8 Octo