mcp

# Spicrawl MCP Server

The Spicrawl MCP server is a Model Context Protocol server that gives AI agents in Claude Code, Claude Desktop, Cursor, VS Code, Windsurf, Gemini CLI and Codex web scraping to Markdown, structured data extraction, screenshots, browser sessions and batch scraping as 25 native tools.

[![npm version](https://img.shields.io/npm/v/@spicrawl/mcp.svg)](https://www.npmjs.com/package/@spicrawl/mcp)
[![npm downloads](https://img.shields.io/npm/dm/@spicrawl/mcp.svg)](https://www.npmjs.com/package/@spicrawl/mcp)
[![license](https://img.shields.io/npm/l/@spicrawl/mcp.svg)](https://github.com/OfficialSpicrawl/mcp/blob/main/LICENSE)
[![smithery badge](https://smithery.ai/badge/spicrawl/web-scraper)](https://smithery.ai/servers/spicrawl/web-scraper)
[![CI](https://github.com/OfficialSpicrawl/mcp/actions/workflows/ci.yml/badge.svg)](https://github.com/OfficialSpicrawl/mcp/actions/workflows/ci.yml)

**[Docs](https://docs.spicrawl.com/agents/mcp)** · **[Get an API key](https://app.spicrawl.com/signup)** · **[TypeScript SDK](https://github.com/OfficialSpicrawl/sdk)** · **[CLI](https://github.com/OfficialSpicrawl/cli)** · **[Agent plugins](https://github.com/OfficialSpicrawl/agent-plugins)** · **[Changelog](https://github.com/OfficialSpicrawl/mcp/blob/main/CHANGELOG.md)**

<p>
  <a href="https://cursor.com/en/install-mcp?name=spicrawl&config=eyJ1cmwiOiJodHRwczovL21jcC5zcGljcmF3bC5jb20vbWNwIiwiaGVhZGVycyI6eyJBdXRob3JpemF0aW9uIjoiQmVhcmVyICR7ZW52OlNQSUNSQVdMX0FQSV9LRVl9In19"><img src="https://cursor.com/deeplink/mcp-install-dark.svg" alt="Add Spicrawl MCP server to Cursor" height="32" /></a>
  <a href="https://vscode.dev/redirect/mcp/install?name=spicrawl&inputs=%5B%7B%22type%22%3A%22promptString%22%2C%22id%22%3A%22spicrawl-api-key%22%2C%22description%22%3A%22Spicrawl%20API%20key%22%2C%22password%22%3Atrue%7D%5D&config=%7B%22type%22%3A%22http%22%2C%22url%22%3A%22https%3A%2F%2Fmcp.spicrawl.com%2Fmcp%22%2C%22headers%22%3A%7B%22Authorization%22%3A%22Bearer%20%24%7Binput%3Aspicrawl-api-key%7D%22%7D%7D"><img src="https://img.shields.io/badge/VS_Code-Install_Spicrawl_MCP-0098FF?style=for-the-badge&logo=visualstudiocode&logoColor=white" alt="Install Spicrawl MCP server in VS Code" height="32" /></a>
</p>

## Quickstart

1. Create an API key at [app.spicrawl.com](https://app.spicrawl.com/signup). Keys start with `spicrawl_live_` or `spicrawl_test_`.
2. Connect the **hosted endpoint**, `https://mcp.spicrawl.com/mcp` (MCP Streamable HTTP, nothing to install):

   ```bash
   export SPICRAWL_API_KEY=spicrawl_live_...
   claude mcp add --transport http spicrawl https://mcp.spicrawl.com/mcp \
     --header "Authorization: Bearer $SPICRAWL_API_KEY"
   ```

   Codex: `codex mcp add spicrawl --url https://mcp.spicrawl.com/mcp --bearer-token-env-var SPICRAWL_API_KEY`

3. Or run it **locally over stdio** (Node.js 20+), for clients that only launch local servers:

   ```bash
   claude mcp add spicrawl --env SPICRAWL_API_KEY=$SPICRAWL_API_KEY -- npx -y @spicrawl/mcp
   ```

4. Ask your agent: *"Use Spicrawl to read https://example.com/pricing as markdown and list each plan with its price."* It calls `spicrawl_scrape` and gets the page back as Markdown.

Both transports serve the same 25 `spicrawl_*` tools, and every call runs under your key with the same scopes, limits and credits as a direct API call.

## What it does

- **Scrape any URL to Markdown**, text, HTML, a JSON envelope or a printed PDF with one tool, `spicrawl_scrape`. Markdown is the default and strips the page to its main content.
- **Render JavaScript** with `render: true` (3 credits). A plain fetch costs 1 credit, screenshots, PDFs and `actions` use a full browser (8 credits), and `mode: "auto"` escalates from fetch to browser and bills only the rung that worked ([price table](https://docs.spicrawl.com/credits)). You never pick the rendering yourself; Spicrawl chooses the cheapest that can do what the request asks.
- **Extract structured data** with a CSS/XPath selector map (`extract`) or from the JSON-LD, OpenGraph, Twitter Card and microdata a page publishes (`autoparse`).
- **Drive the page** before capture with 8 typed browser actions: `click`, `fill`, `wait_for`, `wait_for_navigation`, `scroll` (including infinite scroll), `select`, `evaluate` and `screenshot`.
- **Return screenshots and PDFs the model can see**: screenshots arrive as MCP image blocks (up to 5 MB of base64 each) and printed pages as `application/pdf` resource blocks (up to 10 MB).
- **Batch scrape thousands of URLs** as one async job, up to 10,000 items per call, with 9 tools to submit, poll, page through results, retry, cancel and keep a job open for crawling.
- **Keep a logged-in browser** with persistent sessions: cookies, storage and a fixed rendering tier reused across scrapes.
- **Answer "what did that cost?"** from usage and request-log tools, and **read the Spicrawl docs** from inside the agent.

## Which tool to use

- If you need one page's content or data now, use `spicrawl_scrape`.
- If the page is blank or a skeleton without JavaScript, call `spicrawl_scrape` again with `render: true`.
- If you know the page's structure, use `extract` with CSS selectors; if you don't, try `autoparse`.
- If you have more than about 20 URLs, use `spicrawl_batch_submit`, then `spicrawl_batch_status` and `spicrawl_batch_results`.
- If you discover URLs as you go (crawling), submit with `open: true`, append with `spicrawl_batch_add_items` and finish with `spicrawl_batch_close`.
- If you must log in first, use `spicrawl_session_create` and pass its `id` as `session_id` to `spicrawl_scrape`.
- If a call failed and you need to know why, use `spicrawl_request_get` with the request id, or `spicrawl_requests_list` with `only_errors`.
- If you are unsure of a parameter or an error code, use `spicrawl_docs_search` before guessing.

## Tools

25 tools, grouped by task. Tools reject unknown arguments, so a misspelt field fails with an error naming it instead of being silently dropped. Every tool sets a `title` and explicit `readOnlyHint`, `destructiveHint` and `openWorldHint` annotations, so a client knows which calls to confirm: `spicrawl_scrape` is not read-only (its `method` and `actions` can submit forms), and `spicrawl_batch_cancel`, `spicrawl_session_release` and `spicrawl_session_delete` are destructive. Every tool also declares an `outputSchema` and returns matching `structuredContent` beside the same text it always returned, so a client can read fields (`credits`, `status`, `next_cursor`, ...) without parsing text. The schemas list what an agent can rely on and stay open: fields the API adds later pass through, and statuses and tiers are plain strings. A tool that returns a document or a JSON array (`spicrawl_docs_read`, `spicrawl_batch_task_content`) puts it under `content` in `structuredContent`. Full argument reference: [docs.spicrawl.com/agents/mcp](https://docs.spicrawl.com/agents/mcp#tools).

### Scrape a website to Markdown

| Tool | What it does | Key parameters |
|---|---|---|
| `spicrawl_scrape` | Retrieve one URL as `markdown` (default), `text`, `html`, `json` or `pdf`, with optional rendering, extraction, links, screenshots and browser actions. | `url`, `format`, `render`, `mode`, `main_content_only`, `include_tags`, `exclude_tags`, `extract`, `autoparse`, `links`, `screenshot`, `actions`, `wait_for`, `session_id`, `proxy`, `cache`, `max_cost` |

### Batch scrape many URLs

| Tool | What it does | Key parameters |
|---|---|---|
| `spicrawl_batch_submit` | Queue many URLs as one async job with shared settings; returns the job with `estimated_credits`. | `urls` or `items`, `render`, `format`, `open`, `max_attempts`, `credit_budget`, `max_cost` |
| `spicrawl_batch_list` | List the project's batch jobs, newest first. | `status`, `limit` (max 200), `cursor` |
| `spicrawl_batch_status` | One job's status and progress. | `job_id` |
| `spicrawl_batch_results` | A page of finished items; works while the job runs. | `job_id`, `status`, `limit` (max 5000), `cursor` |
| `spicrawl_batch_task_content` | The full payload of one item. | `job_id`, `seq` |
| `spicrawl_batch_cancel` | Cancel a job; finished items keep their results. | `job_id` |
| `spicrawl_batch_retry` | Re-queue every failed item. | `job_id` |
| `spicrawl_batch_add_items` | Append URLs to a job submitted with `open: true`. | `job_id`, `urls` or `items` |
| `spicrawl_batch_close` | Stop an open job accepting items so it can complete. | `job_id` |

### Browser sessions

| Tool | What it does | Key parameters |
|---|---|---|
| `spicrawl_session_create` | Create a persistent browser identity (cookies, storage, a fixed rendering tier). | `engine`, `ttl_seconds`, `session_context` |
| `spicrawl_session_list` | List sessions, newest first. | `status`, `engine`, `limit`, `cursor` |
| `spicrawl_session_get` | One session's status, rendering tier, exit, usage and expiry. | `session_id` |
| `spicrawl_session_context` | Dump a live session's cookies and storage (contains secrets). | `session_id` |
| `spicrawl_session_release` | End a session; its record stays as `released`. | `session_id` |
| `spicrawl_session_delete` | Permanently delete a session and its state. | `session_id` |

### Usage and request history

| Tool | What it does | Key parameters |
|---|---|---|
| `spicrawl_usage` | Billing-grade usage over a date window (`read` scope). | `from`, `to`, `group_by`, `metrics` |
| `spicrawl_usage_summary` | Current-period totals (`read` scope). | none |
| `spicrawl_usage_reconciliation` | One day's drift between the request log and the billed rollup (`read` scope). | `day` |
| `spicrawl_requests_list` | Recent API calls with status, error code, rendering tier, latency and credits. | `only_errors`, `status`, `limit`, `all_projects` |
| `spicrawl_request_get` | The full log record of one call. | `id` |

### Docs

| Tool | What it does | Key parameters |
|---|---|---|
| `spicrawl_docs_search` | Search the Spicrawl docs; an error code also links to its en