footnote-mcp

<p align="center">
  <img src=".github/banner.png" alt="footnote-mcp" width="350">
</p>

<!-- mcp-name: io.github.KazKozDev/footnote-mcp -->

An MCP server for source-grounded web research. It searches the web, fetches and
extracts pages, pulls structured data out of tables/files/APIs, and — the part that
sets it apart — **verifies that a claim is actually supported by its source** instead of
trusting a snippet. 45 tools over stdio MCP, driven by any MCP client (Claude Desktop,
Cursor) or by the companion [Scholiast](https://github.com/KazKozDev/scholiast) research agent.

The design priority is *trustworthiness over convenience*: search snippets are treated as
discovery only, every fetched page is cached with provenance, and claims are checked
against the source text before they count. It also degrades gracefully — with no API keys
and no config it still works (scraped search + an automatic headless-browser fallback +
an offline verification heuristic); keys and env vars only make it better.

## Quick start

From [PyPI](https://pypi.org/project/footnote-mcp/) (Python ≥ 3.10):

```bash
pip install footnote-mcp
python -m playwright install chromium   # the headless browser used by the fetch fallback
footnote-mcp                            # start the server (speaks MCP over stdio)
```

Or from source:

```bash
python3 -m venv .venv && source .venv/bin/activate
pip install -e .                        # installs the `footnote-mcp` console script + deps
python -m playwright install chromium   # the headless browser used by the fetch fallback
footnote-mcp                            # start the server (speaks MCP over stdio)
```

`footnote-mcp` now waits for an MCP client on stdio. Point a client at it by dropping this
into its MCP settings (Claude Desktop: `claude_desktop_config.json`; Cursor: `~/.cursor/mcp.json`):

```json
{
  "mcpServers": {
    "footnote": { "command": "footnote-mcp" }
  }
}
```

## Hosted MCP endpoint (Render)

This repository also includes an authenticated Streamable HTTP deployment. The
stdio command above remains the recommended local option; the hosted endpoint is
for MCP clients that support remote servers.

1. Push this repository to GitHub, then create a Render **Blueprint** from it.
   Render reads [`render.yaml`](render.yaml), builds the included Dockerfile
   (including Chromium and Tesseract), and exposes the health check at `/healthz`.
2. Choose a unique service name, then set `FOOTNOTE_MCP_PUBLIC_URL` in Render to
   its exact public origin, for example `https://my-footnote-mcp.onrender.com`.
   `FOOTNOTE_MCP_API_KEY` is generated by the Blueprint; it is the owner key.
   Keep it secret and use it only for administration/testing.
3. Connect an MCP client to `https://my-footnote-mcp.onrender.com/mcp` with:

```http
Authorization: Bearer <FOOTNOTE_MCP_API_KEY>
```

The API key is required: a public, unauthenticated research and browser server
would let strangers consume its outbound traffic and call its tools. The service
also validates its configured public host and browser Origin to protect the MCP
endpoint from DNS rebinding.

Render Free is suitable for demos, not production: it sleeps after 15 minutes of
inactivity, cold starts take about a minute, and both outbound bandwidth and
instance hours are limited. The filesystem is ephemeral, so do not rely on the
source cache or browser profile persisting across restarts.

### Giving access to other people

Never distribute the owner key. Create a separate random key for each person:

```bash
python -m footnote_mcp.keygen
```

Then update `FOOTNOTE_MCP_API_KEYS` in Render's Environment settings and
redeploy. Its value is JSON, where each user has a key and a personal
requests-per-minute limit:

```json
{
  "alice": {"key": "fn_alice_key_here", "rpm": 20},
  "bob": {"key": "fn_bob_key_here", "rpm": 10}
}
```

Give each person only their own value. To revoke access, remove that user from
the JSON and redeploy; the other keys keep working. Limits are held in memory,
which is appropriate for this one-instance Free service and reset on restart.

No API keys are required to start — search falls back to scraping Bing + DuckDuckGo. Add
keys later under `"env"` (see [Search backends](#search-backends)). Pass `--headed` to watch
the browser tier work.

Optional runtime variables are documented in [.env.example](.env.example). Copy it to
`.env` for local shells, or paste selected variables into your MCP client config:

```json
{
  "mcpServers": {
    "footnote": {
      "command": "footnote-mcp",
      "env": {
        "TAVILY_API_KEY": "..."
      }
    }
  }
}
```

To run without installing, straight from the source tree:

```bash
PYTHONPATH=src python -m footnote_mcp
```

## Verifying claims — the differentiator

The reason to use this over a plain search tool is `evidence_entailment` and friends:
they tell a claim a source *supports* from one it does not. `benchmarks/run_benchmark.py`
measures that on a labeled set of claim/source pairs (and demos `corroborate_claim` and
`locate_claim_span`):

```bash
python benchmarks/run_benchmark.py                    # offline heuristic (deterministic)
python benchmarks/run_benchmark.py --backend ollama   # LLM judge (needs ollama)
```

Offline-heuristic result on the labeled set ([benchmarks/REPORT.md](benchmarks/REPORT.md)):

| Set | n | Accuracy | Unsupported-claim catch rate | Precision on "supported" |
|-----|---|----------|------------------------------|--------------------------|
| Data domain (numeric + factual) | 15 | 100% | 100% | 100% |
| Overall (incl. semantic) | 18 | 83% | 78% | 80% |

On its design domain — numeric and factual data claims — the offline heuristic never
blesses an unsupported claim and never misses one. Its blind spot is purely-semantic
negation/paraphrase; for those, `evidence_entailment` with `backend="ollama"` (a local LLM
judge) closes the gap. Run the `--backend ollama` line above to score that path on your own
machine.

## Tools

<details>
<summary><b>Discovery and reading</b> (12 tools)</summary>

| Tool | Description |
|------|-------------|
| `web_search` | Configured SearXNG first, then keyed providers, then scraped Bing + DuckDuckGo. Snippets are discovery only. |
| `web_search_recent` | Search restricted to a recency window (day/week/month/year). |
| `web_deep_search` | Automatically route across web/papers/encyclopedia/GitHub/archive sources, then fetch, extract, rerank, and return source context. |
| `web_read` | Fetch one URL, extract text, classify source quality, persist cache metadata. |
| `papers_search` | Search Crossref and arXiv through one normalized, zero-key paper contract. |
| `encyclopedia_search` | Search Wikipedia/Wikidata entities or run read-only Wikidata SPARQL. |
| `github_search` | Search public repositories, issues, code, or commits; authentication is optional. |
| `archive_search` | Find URL captures through Wayback Machine and Common Crawl, optionally extracting archived text. |
| `web_archive_fetch` | Find the closest Wayback Machine snapshot for a dead/changed URL. |
| `web_fetch_authenticated` | Fetch a page that needs cookies or custom headers. |
| `web_crawl` | Breadth-first crawl from a start URL, on-host by default (≤ 50 pages). |
| `generate_search_queries` | Generate operator queries (`site:`, `filetype:csv`, API/data-table variants). |

</details>

<details>
<summary><b>Structured data</b> (9 tools)</summary>

| Tool | Description |
|------|-------------|
| `web_extract_tables` | Parse HTML tables into `columns`/`rows` with source-URL provenance. |
| `web_detect_downloads` | Detect linked CSV/TSV/XLS/XLSX/PDF/JSON/XML files. |
| `web_parse_file` | Download and parse CSV/TSV/XLS/XLSX/PDF/JSON. |
| `web_fetch_json` | Fetch direct API/JSON endpoints into parsed JSON. |
| `check_date_completeness` | Validate required date coverage (day/week/month). |
| `resolve_units` | Detect currencies, currency pairs, measurement units. |
| `validate_unit_rows` | Reject rows with incompatible units or currency pairs. |
| `reconcile_time_series` | Align series on a key, compute deltas, flag missing keys/outliers. |
| `export_dataset` | Write consolidated rows to a `csv`/`xlsx`/`json` file. |

</details>

<details>
<summary><b>Source quality and verification</b> (8 tools)</summary>

| Tool | Description |
|------|-------------|
| `classify_source` | Classify official / aggregator / blog / forum / interactive / blocked / error. |
| `evidence_entailment` | Strict claim-vs-source checker: `heuristic`, `auto`, `ollama`, optional `local_nli`. |
| `corroborate_claim` | Triangulate a claim across excerpts (corroborated / conflicting / single_source / …). |
| `locate_claim_span` | Locate supporting sentence(s) with char offsets and a containment score. |
| `source_cache_get` / `source_cache_put` | Inspect and write persistent source-cache entries. |
| `build_research_debug_report` | Compact report of queries, URLs, source quality, verification gaps. |
| `startup_health_check` | Check parser, OCR, browser, and cache dependencies. |

</details>

<details>
<summary><b>Controlled extraction recipes</b> (6 tools)</summary>

When generic parsers fail, synthesize a sandboxed parser:

| Tool | Description |
|------|-------------|
| `tool_spec_propose` | Propose a task-specific extraction recipe spec. |
| `tool_code_generate` | Generate a starter `extract(source_text, input_payload)` recipe. |
| `tool_code_validate` | Validate recipe code against a static safety allowlist. |
| `tool_code_run_sandboxed` | Run validated code in a limited subprocess (JSON output only). |
| `tool_promote` | Save a validated recipe as reusable memory (no server edit). |
| `recipe_registry` | Manage promoted recipes: `list` / `get` / `run` / `delete`. |

</details>

<details>
<summary><b>Browser fallback</b> (10 tools)</summary>

A controlled Chromium session for JS-heavy or interactive pages:

| Tool | Description |
|------|-------------|
| `web_navigate` · `web_snapshot` · `web_click