crawlforge-mcp

<p align="center">
  <img src="https://raw.githubusercontent.com/mysleekdesigns/crawlforge-mcp/main/assets/banner.jpg" alt="CrawlForge MCP Server" width="640">
</p>

<p align="center">
  <b>27 web scraping, crawling, deep-research &amp; autonomous-extraction tools for Claude, Cursor &amp; any MCP client.</b><br>
  Clean Markdown &amp; structured JSON from any site. Get started with <b>1,000 free credits</b> — no credit card required.
</p>

<p align="center">
  <a href="https://opensource.org/licenses/MIT"><img src="https://img.shields.io/badge/License-MIT-yellow.svg" alt="License: MIT"></a>
  <a href="https://nodejs.org/"><img src="https://img.shields.io/badge/node-%3E%3D18.0.0-brightgreen" alt="Node.js Version"></a>
  <a href="https://modelcontextprotocol.io/"><img src="https://img.shields.io/badge/MCP-Compatible-blue" alt="MCP Protocol"></a>
  <a href="https://www.npmjs.com/package/crawlforge-mcp-server"><img src="https://img.shields.io/npm/v/crawlforge-mcp-server.svg" alt="npm version"></a>
  <a href="https://www.npmjs.com/package/crawlforge-mcp-server"><img src="https://img.shields.io/npm/dm/crawlforge-mcp-server.svg" alt="npm downloads"></a>
  <a href="https://github.com/mysleekdesigns/crawlforge-mcp/stargazers"><img src="https://img.shields.io/github/stars/mysleekdesigns/crawlforge-mcp?style=social" alt="GitHub stars"></a>
</p>

<p align="center">
  ⭐ <b><a href="https://github.com/mysleekdesigns/crawlforge-mcp">Star us on GitHub</a></b> to follow along — it genuinely helps others discover the project.
</p>

## Table of Contents

- [Why CrawlForge?](#-why-crawlforge)
- [CrawlForge vs. alternatives](#-crawlforge-vs-alternatives)
- [Quick Start (2 Minutes)](#-quick-start-2-minutes)
- [Available Tools](#-available-tools)
- [Pricing](#-pricing)
- [Advanced Configuration](#-advanced-configuration)
- [Usage Examples](#-usage-examples)
- [Security & Privacy](#-security--privacy)
- [Support](#-support)
- [Contributing](#-contributing)

## 🎯 Why CrawlForge?

- **27 MCP-native tools** — scraping, crawling, search, real Google SERP rank tracking, deep research, an autonomous `agent`, a unified multi-format `scrape`, document processing, stealth browsing, and more, callable directly from your AI assistant.
- **Generous free tier** — 1,000 credits to start instantly, no credit card. Credits never expire and roll over month-to-month.
- **Local-LLM by default** — `extract_with_llm` runs against a local **Ollama** model out of the box: no LLM API key, no per-token cost, and your data never leaves your machine. Cloud (OpenAI/Anthropic) is opt-in.
- **LLM-ready output** — clean Markdown, structured JSON (schema-driven), screenshots, links, and metadata from a single fetch.
- **Autonomous `agent`** — describe what you need in natural language; it plans, gathers, and shapes an answer under orchestrator-enforced hard stops (max steps/URLs/wall-clock) — no URLs required.
- **Security-hardened** — SSRF protection on every request, a fail-closed backend allow-list, a vetted action allowlist for browser automation, and per-tool credit gating.
- **Works everywhere MCP does** — Claude Desktop, Claude Code, Cursor, and any other MCP-enabled client, configured in one command.

## 📊 CrawlForge vs. alternatives

| | **CrawlForge MCP** | Firecrawl | Raw scraping API |
|---|:---:|:---:|:---:|
| Native MCP server | ✅ 27 tools | ✅ | ❌ |
| Free tier | ✅ 1,000 credits, rollover | Limited | Varies |
| Self-hosted / local LLM extraction (Ollama) | ✅ default, $0/token | ❌ | ❌ |
| Autonomous agent (no URLs needed) | ✅ `agent` | ✅ | ❌ |
| Deep research with source verification | ✅ `deep_research` | Partial | ❌ |
| Browser automation / actions | ✅ `scrape_with_actions` | ✅ | Varies |
| Stealth / anti-detection engines | ✅ Chromium + Camoufox | ✅ | Add-on |
| Pre-built site templates | ✅ 10 sites | ❌ | ❌ |
| License | MIT | AGPL-3.0 | Proprietary |

> Comparison reflects publicly documented capabilities at time of writing. CrawlForge is MIT-licensed and MCP-first — built to plug straight into AI coding assistants.

## 🚀 Quick Start (2 Minutes)

### 1. Install from NPM

```bash
npm install -g crawlforge-mcp-server
```

### 2. Setup Your API Key (required)

Every tool requires a CrawlForge API key — new accounts get 1,000 free trial credits to start:

```bash
npx crawlforge-setup
```

This will:
- Guide you through getting your free API key
- Configure your credentials securely
- **Auto-configure Claude Code and Cursor** (if installed)
- Verify your setup is working

**Don't have an API key?** Get one free at [https://www.crawlforge.dev/signup](https://www.crawlforge.dev/signup)

> **One-step setup (v4.6.0+):** `crawlforge init` detects your API key, installs the agent skill, and idempotently merges the MCP config stanza into Claude Code, Claude Desktop, and Cursor. Use `crawlforge init --all --yes` to configure every detected client non-interactively.

### 3. Configure Your IDE (if not auto-configured)

<details>
<summary>🤖 For Claude Desktop</summary>

Add to `claude_desktop_config.json`:
```json
{
  "mcpServers": {
    "crawlforge": {
      "command": "npx",
      "args": ["-y", "crawlforge-mcp-server"]
    }
  }
}
```

**Location:**
- macOS: `~/Library/Application Support/Claude/claude_desktop_config.json`
- Windows: `%APPDATA%/Claude/claude_desktop_config.json`
- Linux: `~/.config/Claude/claude_desktop_config.json`

Restart Claude Desktop to activate.
</details>

<details>
<summary>🖥️ For Claude Code CLI (Auto-configured)</summary>

The setup wizard automatically configures Claude Code by adding to `~/.claude.json`:
```json
{
  "mcpServers": {
    "crawlforge": {
      "type": "stdio",
      "command": "crawlforge-mcp"
    }
  }
}
```

After setup, restart Claude Code to activate.
</details>

<details>
<summary>💻 For Cursor IDE (Auto-configured)</summary>

The setup wizard automatically configures Cursor by adding to `~/.cursor/mcp.json`:
```json
{
  "mcpServers": {
    "crawlforge": {
      "type": "stdio",
      "command": "crawlforge-mcp"
    }
  }
}
```

Restart Cursor to activate.
</details>

> **Which launch command?** `npx -y crawlforge-mcp-server` needs no global install and always runs the published version (recommended for Claude Desktop). For a global install (`npm i -g crawlforge-mcp-server`), use the dedicated `crawlforge-mcp` bin — it resolves on your `PATH`, so it survives Node/nvm version switches. The bare `crawlforge` command still launches the server when an MCP client spawns it over stdio (backward compatibility for configs created before v4.2.5); interactively it's the CLI — run `crawlforge mcp` to start the server by hand.

## 📊 Available Tools

CrawlForge requires a CrawlForge API key — **every tool is metered and consumes credits**. New accounts get **1,000 free trial credits** to start. Get a key at [crawlforge.dev/signup](https://www.crawlforge.dev/signup).

**All Tools** (API key required)

| Tool | Credits | What it does |
|------|---------|--------------|
| `fetch_url` | 1 | Fetch content from any URL |
| `extract_text` | 1 | Extract clean text from web pages |
| `extract_links` | 1 | Get all links from a page |
| `extract_metadata` | 1 | Extract page metadata (title, OG tags, schema.org) |
| `scrape_template` | 1 | Structured data from well-known sites (Amazon, GitHub, LinkedIn, YouTube, Reddit, Hacker News, npm, and more) without writing selectors |
| `list_ollama_models` | 1 | List the Ollama models installed locally (helps you pick a `model` for `extract_with_llm`) |
| `get_batch_results` | 1 | Retrieve paginated results for a `batch_scrape` job by `batchId` |
| `scrape` | 2 | **Unified single-fetch, multi-format extraction.** Pass a `formats` array (markdown/html/rawHtml/text/links/metadata/screenshot/json-schema) plus `onlyMainContent`; one fetch serves every requested format with per-format partial-success warnings |
| `scrape_structured` | 2 | Extract structured data with CSS selectors |
| `extract_content` | 2 | Enhanced content extraction |
| `map_site` | 2 | Discover and map website structure (optional `search=` ranks the discovered URLs) |
| `process_document` | 2 | Multi-format document processing |
| `localization` | 2 | Multi-language and geo-location management |
| `track_changes` | 3 | Monitor content changes over time |
| `analyze_content` | 3 | Comprehensive content analysis |
| `extract_structured` | 3 | LLM-powered schema-driven extraction (your own LLM key or local Ollama) |
| `extract_with_llm` | 3 | Natural-language extraction. Defaults to a local Ollama model; pass `provider: "openai" \| "anthropic"` with the matching key for cloud models (external LLM billed by your provider) |
| `summarize_content` | 4 | Generate intelligent summaries |
| `crawl_deep` | 4 | Deep crawl entire websites |
| `search_web` | 5 | Search the web using Google Search API |
| `serp_rank` | 5 | Check where a domain ranks in Google's **real organic SERP** for a keyword (the position `search_web` can't give). Powered by DataForSEO (`DATAFORSEO_LOGIN`/`DATAFORSEO_PASSWORD`, billed to your own DataForSEO account). Returns `{ configured:false }` and charges **0** credits until configured |
| `batch_scrape` | 5 | Process multiple URLs simultaneously |
| `scrape_with_actions` | 5 | Browser automation chains |
| `generate_llms_txt` | 5 | Generate AI interaction guidelines |
| `stealth_mode` | 5 | Anti-detection browser management |
| `agent` | 8 | **Autonomous research/extraction from a natural-language prompt — no URLs required.** Plans, gathers, and shapes an answer under hard safety stops (max steps/URLs/wall-clock enforced by the orchestrator, never the LLM) |
| `deep_research` | 10 | Multi-stage research with source verification |

For the full canonical capabilities reference (all tools, CLI commands, stealth engines, research workflow), see [SKILL.md](docs/SKILL.md).

<p align="right"><a href="#table-of-contents">↑ Back to top</a></p>

## 💳 Pricing

**Every tool is metered and requires an API key.** New accoun