mcp-search

# search — web search as a module

Five capabilities behind one MCP server: **find** (`web_search`), **read**
(`web_read`), **find images** (`web_image_search`), **screenshot a page**
(`web_screenshot`), and **answer a question from several sources**
(`web_deep_search`).

The module keeps no index of its own. It queries other people's search engines
through a metasearch container, and everything it adds is about one problem:

> **A search tool fails in ways that look exactly like success.** An engine
> answers with somebody else's subject; a page returns text that is an anti-bot
> shield; a corpus of sixteen sources turns out to be two engines counted eight
> times. This module's job is to make those cases *distinguishable*, and to say
> so in fields you can branch on.

**Search reads by default.** The top three results come back with their text in
`content`, so one call answers a question instead of handing over links. The
cheap path is still there: `read: false` returns links alone in a fraction of a
second.

Three of the tools are easy to confuse, so the line is drawn explicitly:
`web_search` and `web_read` return **material**; `web_deep_search` returns a
**judgement** — it composes its own queries, goes in waves, and says what it
failed to find.

## Where to go next

* **[HOWTO-CALL.md](HOWTO-CALL.md)** — how to call it and, more importantly, how
  to read what comes back. Start here.
* **[ALGORITHM.md](ALGORITHM.md)** — what happens, step by step, when a request
  arrives.
* **[manifest.yaml](manifest.yaml)** — the machine-readable description.
* **contracts/** — one contract per capability: what is promised, what is not,
  and what the failure directions are.
* **[measures/](measures/)** — the numbers: which engines are alive, what the
  module withstands under load, what changing transport bought.

## Install

From an open repository page to a working answer. Nothing is assumed to be on
your disk already:

```bash
git clone https://github.com/AG-Bureau/mcp-search
cd mcp-search
cp .env.example .env
echo "SEARXNG_SECRET=$(openssl rand -hex 32)" >> .env
docker compose -f docker-compose.yml -f wiring/expose-localhost.yml up -d --build
curl -s http://127.0.0.1:8081/healthz
curl -s -X POST http://127.0.0.1:8081/mcp \
     -H 'Content-Type: application/json' \
     -d '{"jsonrpc":"2.0","id":1,"method":"tools/list"}'
```

The fourth line is not decoration. Without a value in `SEARXNG_SECRET` the very
next command refuses — `required variable SEARXNG_SECRET is missing a value` —
and that refusal is deliberate (see below). `openssl rand -hex 32` is one way to
satisfy it; any long random string that is not taken from somebody's history
will do.

**`SEARXNG_SECRET` is mandatory and `up` refuses without it.** That is
deliberate: with no key of its own the metasearch does not fail — it comes up
with a publicly known one from its image template, silently.

**A model key is NOT mandatory.** Search, image search and screenshots never
call a model; reading calls one on a single branch — a PDF with no text layer,
which has to be recognised. Without a key that branch and deep search declare
themselves unconfigured and name the missing variable rather than returning a
quiet nothing.

**The protocol is OpenAI-compatible, and it is verified on GLM.** Providers
differ in details beyond the common part, so the differences are carried in
variables rather than in code — `LLM_API_BASE`, the two model names, and
`LLM_DISABLE_THINKING`. If your provider refuses a request, look there first: the
dialect field is not sent by default precisely because a strict provider answers
`400` to it.

**Set `LLM_API_BASE` in full, and verify it.** The obvious guess at a provider's
address can answer `429: Insufficient balance` because the subscription lives on
a different path of the same domain. The module behaves correctly — it returns
the provider's answer verbatim — but from outside that reads as a broken product.
Take the address from your provider's documentation, not by analogy.

**`--build` is in the command on purpose.** The services carry both `build:` and
`image:`; with an image of the same tag already present, compose reuses it and
builds nothing. On a clean machine there is no image, but the flag keeps an
update from raising yesterday's build.

The overlay publishes the port **on loopback only**, and it is the only
publishing overlay that ships. Publishing more widely is two lines of your own,
and the checks to run before them are listed in
`wiring/expose-localhost.yml` — a published container port does not go through
the host firewall's usual chain, so an empty `DOCKER-USER` means the firewall
does not apply to it at all while still reporting that it does.

## Cost, and the knobs that control it

**Three capabilities out of five never call a model.** Search, image search and
screenshots are HTTP requests: they spend no tokens at all. Reading spends tokens
on ONE branch — a PDF with no text layer, recognised by a vision model — and is
free on every other. The fifth, `web_deep_search`, always calls a model. Without
a key the two model-dependent paths declare themselves unconfigured rather than
failing.

That is the difference from a search built into a model. A built-in tool has no
knobs: it always works in one mode, always returns its own volume of text into
the context, and that text is always paid for in tokens.

Here the volume is an argument. One and the same query, measured on one machine:

| call | returned | time | model calls |
|---|---|---|---|
| `read: false`, `max_results: 6` | 3 844 characters | 0.6 s | 0 |
| `read_top: 1`, `max_results: 3` | 8 833 characters | 1.6 s | 0 |
| `read_top: 3`, `max_results: 6` | 9 805 characters | 6.3 s | 0 |
| `web_deep_search` | a full digest | 36 s | 6 |

A single argument moves the volume by an order of magnitude and the time by a
factor of sixty. The knobs are `read`, `read_top`, `max_results`, `min_engines`
and `per_engine`; `min_engines` is the one that buys independence and the one
that multiplies outbound requests proportionally.

*(One machine, one query. Take the shape, not the digits.)*

**And the honest other half.** On a well-covered question the module is SLOWER
than a search built into a model and returns the same answer. Measured on a
question about a large company's annual revenue: 36 seconds for a deep search
against seconds for a built-in one, and the same figure in both.

The gain begins where the answer is not on the surface, or where you need to know
what to trust: which engine found a link, whether the sources agree, whether the
page was a page or an anti-bot wall, whether one name covers several different
subjects. A built-in search answers the question; this one also answers how much
the answer is worth.

## What it is made of

**A metasearch container** — somebody else's open service. It holds the result
parsers for dozens of engines, maintained by their community rather than by us.
That is exactly why it is here: our own parsers would need repairing after every
redesign somebody else ships.

**The adapter** — our process, and the module proper: from outside, only this is
visible. It holds the doors over ONE implementation of each capability — MCP for
tool clients, plain HTTP contracts, a health check for orchestration, and the
observation views `/engines` and `/pages`. A second implementation "for another
protocol" would diverge from the first at the first edit.

**The reader** — part of the adapter, in a file of its own, because the reading
policy (per-domain pacing, stub rejection, the chunked cursor) belongs beside the
measurements that explain it. HTML is parsed on the standard library: the price
is named honestly — tables come out as lines rather than tables — and in exchange
there is one less library to update on the path where every outside page arrives.
PDFs are another matter: a stdlib parser was written, measured and rejected, so
the image carries four libraries and 419 MB, each argued for in
`adapter/Dockerfile`.

**The prober** — our process. Continuously, one engine at a time, it asks every
engine the metasearch knows and writes a verdict to a database. It exists because
the set of living engines changes within hours, and a point measurement cannot
see that: an engine that was the best of the set returns nothing weeks later
without saying a word about it.

The same prober measures **reading references** — nine pages whose content is
known in advance, each covering its own defect class. Two of them are
**negative**: addresses that certainly do not exist, from which the expected
verdict is `stub`. Without them the shield detector would degrade unnoticed,
because it always says `clean`.

## How engines are chosen

**One engine at a time, not a fan-out.** The order is fixed; we walk it top down
and stop as soon as there is enough. If an engine refuses, the next one goes —
switching is not a separate mechanism but a property of the order. A fan-out
costs several times the requests to other people's services and brings a block
closer, while adding almost nothing on an ordinary query: different engines
overlap heavily on the same question.

**At most one request per engine per interval.** The metasearch has no rate
regulator at all, so the module holds one. An engine that may not be asked right
now is skipped rather than waited for. If nobody may be asked, the answer is a
refusal with a reason — "we asked nobody" must be distinguishable from "nothing
was found".

**The pool is COMPUTED from observation, not written by hand.** This is the main
rule here. A hand-written list of engines needs revising as often as the engines
change — three times in one day is not unusual, each revision against the
previous one and each correct on the data available. The problem is neither the
engines nor the quality of the decisions: a decision freezes while observation
goes on.

So the pool is the best-N by reference hit share, recomputed continuously.
**No entry threshold, no