mcp-server-web-fetcher

# mcp-server-web-fetcher

[![CI](https://github.com/vojtisprime11/mcp-server-web-fetcher/actions/workflows/ci.yml/badge.svg)](https://github.com/vojtisprime11/mcp-server-web-fetcher/actions/workflows/ci.yml)
[![npm version](https://img.shields.io/npm/v/mcp-server-web-fetcher.svg)](https://www.npmjs.com/package/mcp-server-web-fetcher)
[![License: MIT](https://img.shields.io/badge/license-MIT-yellow.svg)](LICENSE)
[![TypeScript](https://img.shields.io/badge/TypeScript-strict-3178c6.svg)](tsconfig.json)
[![Node.js](https://img.shields.io/badge/node-%3E%3D20.18.1-5fa04e.svg)](https://nodejs.org)
[![MCP](https://img.shields.io/badge/MCP-1.30-6b4fbb.svg)](https://modelcontextprotocol.io)
[![MCP Registry](https://img.shields.io/badge/MCP%20Registry-listed-6b4fbb.svg)](https://registry.modelcontextprotocol.io/)

A fast, dependency-light [Model Context Protocol](https://modelcontextprotocol.io) server that turns
any web page into something a language model can actually read: clean Markdown, structured metadata,
and a classified list of links.

Web pages are 90% chrome. Scripts, cookie banners, navigation, sidebars and tracking pixels burn
context and derail reasoning. This server strips all of that on the way in, so the model sees the
article and nothing else.

```
┌────────────┐   stdio/JSON-RPC   ┌──────────────────────┐   HTTPS   ┌──────────┐
│ MCP client │ ─────────────────► │ mcp-server-web-      │ ────────► │ web page │
│ (Claude,   │ ◄───────────────── │ fetcher              │ ◄──────── │          │
│  Kiro, …)  │  Markdown + JSON   │ fetch → clean → md   │           └──────────┘
└────────────┘                    └──────────────────────┘
```

## Features

- **Clean Markdown output.** Readability-style main-content detection, GFM tables, language-tagged
  code fences, absolute links. No leftover HTML, even for gnarly nested tables.
- **Context-window aware.** Long pages are paginated with `startIndex` / `nextStartIndex` instead of
  being silently cut in half.
- **Structured metadata.** Title, description, canonical, `lang`, author, publish/modify dates,
  Open Graph, Twitter cards, JSON-LD, `hreflang` alternates, RSS/Atom feeds, h1–h6 outline and raw
  HTTP headers.
- **Link intelligence.** Absolute URLs, anchor text, `rel`, `nofollow`, internal vs external
  classification, scope filters, de-duplication.
- **SSRF hardening.** Loopback, private, link-local and cloud-metadata addresses are blocked on
  _every_ redirect hop, not just the first request. Credentials in URLs are stripped.
- **Predictable failures.** Every error carries a stable machine-readable `code`
  (`TIMEOUT`, `HTTP_ERROR`, `BLOCKED_HOST`, `RESPONSE_TOO_LARGE`, …), a retryability flag and a
  recovery hint, so the model can self-correct instead of guessing.
- **Typed end to end.** Strict Zod input schemas plus published `outputSchema`, so clients get
  validated `structuredContent`, not prose they have to re-parse.
- **Well behaved.** Byte caps, timeouts, redirect limits, charset sniffing (including legacy
  encodings), a small TTL cache, transient-failure retries and optional `robots.txt` enforcement.
- **115 tests, no network required.** Vitest unit tests plus in-memory MCP protocol tests.

## Quickstart

Requires Node.js 20.18.1 or newer (inherited from cheerio, which needs undici 7).

```bash
# run it without installing
npx -y mcp-server-web-fetcher

# or install globally
npm install -g mcp-server-web-fetcher
mcp-server-web-fetcher
```

The server speaks MCP over stdio, so on its own it just waits for a client. Point a client at it:

### Claude Desktop

`~/Library/Application Support/Claude/claude_desktop_config.json` on macOS,
`%APPDATA%\Claude\claude_desktop_config.json` on Windows:

```json
{
  "mcpServers": {
    "web-fetcher": {
      "command": "npx",
      "args": ["-y", "mcp-server-web-fetcher"]
    }
  }
}
```

With configuration, using a global install:

```json
{
  "mcpServers": {
    "web-fetcher": {
      "command": "mcp-server-web-fetcher",
      "env": {
        "WEB_FETCHER_TIMEOUT_MS": "20000",
        "WEB_FETCHER_RESPECT_ROBOTS": "true"
      }
    }
  }
}
```

Restart Claude Desktop, then ask it to summarise a URL.

### Claude Code

```bash
claude mcp add web-fetcher -- npx -y mcp-server-web-fetcher
```

### Kiro, Cursor, Windsurf and other MCP clients

Same shape, in `.kiro/settings/mcp.json` / the client's MCP config file:

```json
{
  "mcpServers": {
    "web-fetcher": {
      "command": "npx",
      "args": ["-y", "mcp-server-web-fetcher"],
      "disabled": false,
      "autoApprove": ["fetch_page_markdown", "extract_metadata", "extract_links"]
    }
  }
}
```

### MCP Registry

The server is listed in the [official MCP Registry](https://registry.modelcontextprotocol.io/)
as `io.github.vojtisprime11/web-fetcher`, so clients that read the registry can discover it
directly:

```bash
curl "https://registry.modelcontextprotocol.io/v0.1/servers?search=io.github.vojtisprime11/web-fetcher"
```

### From source

```bash
git clone https://github.com/vojtisprime11/mcp-server-web-fetcher.git
cd mcp-server-web-fetcher
npm install
npm run build
node dist/index.js          # or: npm run inspect
```

```json
{
  "mcpServers": {
    "web-fetcher": {
      "command": "node",
      "args": ["/absolute/path/to/mcp-server-web-fetcher/dist/index.js"]
    }
  }
}
```

## Tools

All three tools are read-only, idempotent and open-world (annotated as such in the protocol), and
all take an absolute `http(s)` URL.

### `fetch_page_markdown`

Downloads a page and returns clean, LLM-friendly Markdown.

| Parameter         | Type                | Default       | Description                                                    |
| ----------------- | ------------------- | ------------- | -------------------------------------------------------------- |
| `url`             | string              | required      | Absolute http(s) URL.                                          |
| `maxLength`       | integer 500–1000000 | `25000`       | Markdown characters to return per call.                        |
| `startIndex`      | integer ≥ 0         | `0`           | Character offset; use `nextStartIndex` from the previous call. |
| `mainContentOnly` | boolean             | `true`        | Drop nav/header/footer/sidebar, keep the densest block.        |
| `includeLinks`    | boolean             | `true`        | Keep Markdown links (`false` inlines the text only).           |
| `includeImages`   | boolean             | `false`       | Keep images as `![alt](src)`.                                  |
| `includeMetadata` | boolean             | `true`        | Attach a short metadata summary.                               |
| `timeoutMs`       | integer 1000–120000 | env / `15000` | Per-request timeout.                                           |

Request:

```json
{
  "name": "fetch_page_markdown",
  "arguments": {
    "url": "https://example.com/blog/caching",
    "maxLength": 8000,
    "mainContentOnly": true,
    "includeImages": false
  }
}
```

`structuredContent`:

```json
{
  "url": "https://example.com/blog/caching",
  "requestedUrl": "https://example.com/blog/caching",
  "status": 200,
  "contentType": "text/html; charset=utf-8",
  "title": "How caching works",
  "markdown": "# How caching works\n\nCaching is the art of **not** doing work twice...",
  "markdownLength": 7984,
  "totalLength": 21874,
  "startIndex": 0,
  "endIndex": 7984,
  "nextStartIndex": 7984,
  "truncated": true,
  "wordCount": 1203,
  "bytesDownloaded": 148213,
  "elapsedMs": 412,
  "fromCache": false,
  "redirects": [],
  "metadata": {
    "title": "How caching works",
    "description": "A deep dive into HTTP caching.",
    "canonical": "https://example.com/blog/caching",
    "language": "en",
    "author": "Ada Lovelace",
    "publishedTime": "2026-01-15T09:00:00Z"
  }
}
```

To read the rest, call again with `"startIndex": 7984`. Keep going while `nextStartIndex` is not
`null`.

### `extract_metadata`

Everything a model needs to classify a page, without spending context on its body.

| Parameter            | Type                | Default       | Description                                                 |
| -------------------- | ------------------- | ------------- | ----------------------------------------------------------- |
| `url`                | string              | required      | Absolute http(s) URL.                                       |
| `includeJsonLd`      | boolean             | `true`        | Include parsed JSON-LD blocks (malformed ones are skipped). |
| `includeHeadings`    | boolean             | `true`        | Include the h1–h6 outline.                                  |
| `includeHttpHeaders` | boolean             | `true`        | Include response headers, lower-cased.                      |
| `timeoutMs`          | integer 1000–120000 | env / `15000` | Per-request timeout.                                        |

```json
{
  "name": "extract_metadata",
  "arguments": { "url": "https://example.com/blog/caching", "includeJsonLd": true }
}
```

`structuredContent` (abridged):

```json
{
  "url": "https://example.com/blog/caching",
  "status": 200,
  "charset": "utf-8",
  "title": "How caching works",
  "description": "A deep dive into HTTP caching.",
  "canonical": "https://example.com/blog/caching",
  "language": "en",
  "author": "Ada Lovelace",
  "publishedTime": "2026-01-15T09:00:00Z",
  "modifiedTime": null,
  "robots": "index, follow",
  "favicon": "https://example.com/favicon.ico",
  "openGraph": {
    "title": "How caching works",
    "type": "article",
    "image": "https://example.com/img/cover.png"
  },
  "twitter": { "card": "summary_large_image", "site": "@example" },
  "alternates": [{ "hreflang": "de", "href": "https://example.com/de/blog/caching" }],
  "feeds": [
    { "title": "Feed", "href": "https://example.com/feed.xml", "type": "application/rss+xml" }
  ],
  "jsonLd": [{ "@type": "Article", "headline": "How caching works" }],
  "h