web2md-mcp

# Web2MD MCP server

An MCP server that turns any web page into clean Markdown, plain text or token-bounded RAG chunks, using the Web2MD API on RapidAPI.

[![npm](https://img.shields.io/npm/v/web2md-rag-mcp)](https://www.npmjs.com/package/web2md-rag-mcp)
[![CI](https://github.com/kanavsingla28/web2md-mcp/actions/workflows/ci.yml/badge.svg)](https://github.com/kanavsingla28/web2md-mcp/actions/workflows/ci.yml)

## What it does

- `read_url`: fetch a web page and return its main content as Markdown or plain text, without nav, ads or boilerplate.
- `chunk_url`: fetch a web page and split it into heading-aware chunks sized in tokens, for RAG and embeddings.
- `convert_html`: convert raw HTML you already have into Markdown or plain text.
- `page_metadata`: get a page's title, author, published date, description and similar fields without the full content.

## Setup in 3 steps

1. Subscribe to Web2MD on [RapidAPI](https://rapidapi.com/kanavsingla28/api/web2md-webpage-to-markdown-rag-chunks). The BASIC plan is free and includes 500 requests per month.
2. Copy your `X-RapidAPI-Key` from the Endpoints tab of the listing.
3. Add the server to your client.

Requires Node.js 22 or newer.

Claude Code:

```bash
claude mcp add web2md -e RAPIDAPI_KEY=your-key -- npx -y web2md-rag-mcp
```

Claude Desktop (`claude_desktop_config.json`) and Cursor (`~/.cursor/mcp.json`) use the same JSON:

```json
{
  "mcpServers": {
    "web2md": {
      "command": "npx",
      "args": ["-y", "web2md-rag-mcp"],
      "env": { "RAPIDAPI_KEY": "your-key" }
    }
  }
}
```

Windsurf: put the same block in `~/.codeium/windsurf/mcp_config.json`.

## Tools

| Tool | What it does | Arguments | Returns |
| --- | --- | --- | --- |
| `read_url` | Fetch a page as clean Markdown or text | `url` (up to 2048 chars); `format`: `markdown`, `text` or `json`; `include_links`, `include_images`, `include_tables` | Title, token count and the content; `format: "json"` returns the full JSON document |
| `chunk_url` | Split a page into RAG chunks | `url`; `max_tokens` 64-8192 (default 512); `overlap_tokens` 0-1024 (default 50); `include_links`, `include_images`, `include_tables` | JSON with `total_tokens` and `chunks`, each with its heading path and token count |
| `convert_html` | Convert raw HTML | `html` (up to 5,000,000 chars); optional `url` to resolve relative links; `format`; `include_links`, `include_images`, `include_tables` | Same as `read_url` |
| `page_metadata` | Get page metadata | `url` | JSON with title, author, date, description, site name, language, canonical URL, image, word count, reading time |

## Limits and plans

| Plan | Price | Requests per month |
| --- | --- | --- |
| BASIC | $0 | 500 |
| PRO | $9 | 10,000 |
| ULTRA | $29 | 60,000 |
| MEGA | $79 | 250,000 |

- RapidAPI rate limit: 1,000 requests per hour.
- Pages up to 5 MB.
- robots.txt is respected.
- JavaScript-heavy pages may return `no_content`. Rendering support is planned.

## Errors

Tool errors look like `Web2MD error [code]: message`. Input-validation failures (a bad URL, out-of-range numbers) come from the MCP SDK as `MCP error -32602: Input validation error ...` instead.

| Code | Meaning |
| --- | --- |
| `missing_key` | `RAPIDAPI_KEY` is not set |
| `invalid_key` | `RAPIDAPI_KEY` contains characters not allowed in an HTTP header, often a pasted newline |
| `rapidapi_auth` | RapidAPI rejected the key, or you are not subscribed |
| `rapidapi_quota` | Your plan's quota or rate limit is used up |
| `invalid_request` | The request failed validation (HTTP 422) |
| `invalid_url` | 400: not http/https, has credentials, no host, bad port, or longer than 2048 chars |
| `blocked_destination` | 400: host resolves to a private, loopback, link-local or metadata address; port not 80/443; localhost/.local/.internal. Applies to every redirect |
| `dns_error` | 400: the host does not resolve |
| `robots_disallowed` | 403: the site's robots.txt disallows the `web2md` user agent for that path |
| `page_too_large` | 413: page larger than 5 MB |
| `unsupported_content_type` | 415: not HTML |
| `no_content` | 422: fewer than 10 words of main content, often a JavaScript-only page |
| `host_rate_limited` | 429: per-target-site limit that protects the sites fetched. It is not your quota |
| `target_blocked` | 502: the site answered 401/403/429/503, often a bot wall |
| `target_error` | 502: the site answered another 4xx/5xx |
| `fetch_failed` | 502: could not connect, or TLS failed |
| `too_many_redirects` | 502: more than 5 redirects |
| `fetch_timeout` | 504: the fetch took more than 15 s |
| `timeout` | The Web2MD request timed out |
| `network_error` | The request could not reach the API |
| `bad_response` | The API returned a non-JSON success response |
| `http_<status>` | Any other HTTP failure, with a short snippet of the response |
| `internal_error` | An unexpected error inside the MCP server |

## Privacy

Your key is sent only to the RapidAPI gateway (`web2md-webpage-to-markdown-rag-chunks.p.rapidapi.com`) and is never logged.

## Development

```bash
npm ci
npm test
npm run lint
npm run build
RAPIDAPI_KEY=your-key npm run smoke
```

## Licence

MIT