web-crawler-mcp-server

# Web Crawler MCP Server (SEO Crawler) by Insightful Pipe

[![MCP Compatible](https://img.shields.io/badge/MCP-Compatible-blue)](https://insightfulpipe.com/mcp-servers/crawler)
[![Insightful Pipe](https://img.shields.io/badge/Insightful_Pipe-MCP_Servers-purple)](https://insightfulpipe.com/mcp-servers)
[![License](https://img.shields.io/badge/License-MIT-green.svg)](LICENSE)

> **Comprehensive SEO auditing and web crawling through AI with MCP.**

Part of the [Insightful Pipe MCP Server Collection](https://insightfulpipe.com/mcp-servers) — The Web Crawler MCP server enables Claude, ChatGPT, Cursor, and other AI assistants to perform SEO audits, extract data, and analyze websites. Run 25+ specialized SEO checks, generate sitemaps, and extract brand information.

[![Explore All MCP Servers](https://img.shields.io/badge/Explore_All-MCP_Servers-blue?style=for-the-badge)](https://insightfulpipe.com/mcp-servers)

![Web Crawler MCP Server](https://insightfulpipe.com/images/ip-logo.png)

## MCP Server URL

```
https://crawler.insightfulmcp.com/
```

## What is Web Crawler MCP?

Web Crawler MCP is a **remote Model Context Protocol server** that provides AI assistants with comprehensive SEO auditing and web crawling capabilities. This integration allows you to:

- Run 25+ specialized SEO checkers on any URL
- Generate XML sitemaps and robots.txt files
- Extract brand design elements from websites
- Audit technical SEO, content quality, and security
- Analyze mobile responsiveness and page speed

## Installation

### Claude

1. Copy the MCP Server URL: `https://crawler.insightfulmcp.com/`
2. Open [Claude Connectors Settings](https://claude.ai/settings/connectors)
3. Scroll to the bottom and click **Add custom connector**
4. Paste the URL and click **Add**
5. Click **Connect** and sign in to InsightfulPipe if asked. Claude then lists the connector as connected.

### ChatGPT

Custom MCP servers are added through ChatGPT's **Developer mode**. Availability depends on your ChatGPT plan, and workspace admins may need to allow it.

1. Turn on **Developer mode** in ChatGPT settings
2. Create a new app for a remote MCP server and paste the URL: `https://crawler.insightfulmcp.com/`
3. Sign in to InsightfulPipe if asked. The connection finishes as soon as you are signed in.

See OpenAI's guide: [Developer mode and MCP apps in ChatGPT](https://help.openai.com/en/articles/12584461-developer-mode-and-mcp-apps-in-chatgpt)

### Claude Code

```bash
claude mcp add --transport http crawler https://crawler.insightfulmcp.com/
```

### Cursor

Add the server to `~/.cursor/mcp.json` (all projects) or `.cursor/mcp.json` (one project):

```json
{
  "mcpServers": {
    "crawler": {
      "url": "https://crawler.insightfulmcp.com/"
    }
  }
}
```

When Cursor shows **Needs authentication**, click **Connect** and sign in to InsightfulPipe if asked.

## Available Actions

33 actions: 33 read, 0 write.

### Read Actions (33)

<details>
<summary>Show all 33 read actions</summary>

| Action | Description |
|--------|-------------|
| `ai-crawler-checker` | Check permissions for AI training bots and crawlers |
| `amp-checker` | Verify AMP implementation quality and discoverability |
| `brand-extractor` | Extract brand design details from a website (colors, fonts, buttons, logos) |
| `canonical-checker` | Validate canonical URLs and conflicting hints |
| `character-encoding-checker` | Validate declared charset and encoding consistency |
| `content-optimizer-checker` | Analyze on-page content quality and keyword density |
| `content-quality-checker` | Score readability, word count, and duplication risk |
| `deprecated-html-checker` | Detect obsolete tags or attributes still in use |
| `favicon-checker` | Confirm favicon/icon discovery across devices |
| `headings-checker` | Review heading hierarchy (H1-H6) for structure gaps |
| `hreflang-checker` | Review international hreflang annotations |
| `image-optimization-checker` | Check alt text, formats, and compression levels |
| `indexability-checker` | Review robots/meta directives affecting indexation |
| `internal-linking-checker` | Map internal links and surface opportunities |
| `language-checker` | Verify html[lang] attributes and language targeting |
| `link-analysis-checker` | Detect broken or low-quality links on page |
| `link-extractor` | Extract all links from a page and categorize them (internal and external) |
| `llms-txt-checker` | Parse llms.txt instructions for AI crawlers |
| `meta-tags-checker` | Inspect meta title/description tags across the page |
| `mobile-responsive-checker` | Audit responsive layout rules and breakpoints |
| `mobile-viewport-checker` | Ensure viewport meta tags and responsive meta configs are in place |
| `page-load-checker` | Measure page weight and render-blocking assets |
| `page-speed-checker` | Measure load times and highlight slow resources |
| `robots-txt-checker` | Parse robots.txt directives and highlight blocking issues |
| `robots-txt-generator` | Generate robots.txt content with optional AI bot restrictions |
| `security-headers-checker` | Check CSP/HSTS and other security headers |
| `seo-audit-checker` | Comprehensive SEO audit covering technical and content issues |
| `sitemap-generator` | Crawl the target domain and produce an XML sitemap |
| `social-media-checker` | Validate Open Graph and Twitter Card metadata |
| `ssl-certificate-checker` | Inspect certificate chain, expiry, and HTTPS enforcement |
| `status-code-checker` | Trace HTTP status/redirect chains for a URL |
| `structured-data-checker` | Validate Schema.org markup and structured data coverage |
| `url-structure-checker` | Evaluate keyword usage and cleanliness of URLs |

</details>

## Control What Your AI Can Do

You decide what AI agents can do with each connected account:

- **Turn individual actions on or off** for every connected account, so agents only see the actions you allow.
- **Connect as Read-only or Read & Write.** A read-only connection can only enable read actions.
- **Destructive actions stay off by default.** Actions such as deletes are disabled until an admin enables them.
- **Team access per account.** Restricted team members only use the accounts they are granted, with the read actions enabled on them.

## Usage Examples

### Full SEO Audit

```
"Run a comprehensive SEO audit on https://example.com"
```

### Check Technical SEO

```
"Check the robots.txt and indexability of https://example.com"
```

### Analyze Internal Links

```
"Map all the internal links on this page"
```

### Generate Sitemap

```
"Generate an XML sitemap for https://example.com"
```

### Extract Brand Elements

```
"Extract the brand colors, fonts, and logos from https://company.com"
```

### Security Audit

```
"Check the SSL certificate and security headers for this site"
```

## Why Web Crawler MCP?

### For SEO Professionals
- **Comprehensive audits** - 25+ specialized checks
- **Technical SEO** - Robots, canonical, structured data
- **Content analysis** - Quality scoring and optimization

### For Web Developers
- **Debug issues** - Find broken links, redirect chains
- **Performance** - Page speed and resource analysis
- **Security** - SSL and header validation

### For Agencies
- **Client audits** - Quick site assessments
- **Competitive analysis** - Analyze competitor sites
- **Reporting** - Generate findings conversationally

## Security & Privacy

- **Secure requests** - HTTPS support

## Pricing

The Web Crawler MCP server is included in every InsightfulPipe plan, together with all other MCP servers and the CLI. Plans start at $29.99/month, and you can try it for 7 days. See [insightfulpipe.com/pricing](https://insightfulpipe.com/pricing) for current plans.

## Ready-Made Skills and Prompts

- [Claude skills for SEO and marketing](https://insightfulpipe.com/marketing-claude-skills/marketing) — ready-made skills that run on your connected data
- [Comprehensive SEO Site Audit](https://insightfulpipe.com/marketing-prompts-library/crawler-comprehensive-seo-site-audit)
- [Technical SEO Health Check](https://insightfulpipe.com/marketing-prompts-library/crawler-technical-seo-health-check)
- [Content Quality Assessment](https://insightfulpipe.com/marketing-prompts-library/crawler-content-quality-assessment)

## Explore More MCP Servers by Insightful Pipe

Visit **[insightfulpipe.com/mcp-servers](https://insightfulpipe.com/mcp-servers)** to discover our full collection of MCP servers for marketing and analytics.

### SEO & Performance MCP Servers
- [PageSpeed MCP](https://insightfulpipe.com/mcp-servers/pagespeed) - Performance analysis
- [Google Search Console MCP](https://insightfulpipe.com/mcp-servers/google-search-console) - Search analytics
- [Screenshot MCP](https://insightfulpipe.com/mcp-servers/screenshot) - Visual capture
- [Google My Business MCP](https://insightfulpipe.com/mcp-servers/google-my-business) - Local SEO

### Analytics MCP Servers
- [Google Analytics MCP](https://insightfulpipe.com/mcp-servers/google-analytics) - Website analytics
- [Enrichment MCP](https://insightfulpipe.com/mcp-servers/enrichment) - Data enrichment

**[View All MCP Servers →](https://insightfulpipe.com/mcp-servers)**

## Resources

- [Documentation](https://insightfulpipe.com/docs/connectors-web-crawler)
- [Video Tutorial](https://www.youtube.com/playlist?list=PLJNzvjxzI5Xwe__BJJLAelSF0ewO3mEFk)
- [InsightfulPipe Blog](https://insightfulpipe.com/blog)

## Support

- **Documentation**: [insightfulpipe.com/docs](https://insightfulpipe.com/docs)
- **All MCP Servers**: [insightfulpipe.com/mcp-servers](https://insightfulpipe.com/mcp-servers)
- **Email**: support@insightfulpipe.com

---

**[Insightful Pipe](https://insightfulpipe.com)** — AI-powered marketing analytics through MCP servers. [Explore all integrations →](https://insightfulpipe.com/mcp-servers)

## License

MIT License - see [LICENSE](LICENSE) for details.