crawl4ai

# 🚀🤖 Crawl4AI: the open-source web crawler for LLMs and AI agents

<div align="center">

<a href="https://trendshift.io/repositories/11716" target="_blank"><img src="https://trendshift.io/api/badge/repositories/11716" alt="unclecode%2Fcrawl4ai | Trendshift" style="width: 250px; height: 55px;" width="250" height="55"/></a>

[![GitHub Stars](https://img.shields.io/github/stars/unclecode/crawl4ai?style=social)](https://github.com/unclecode/crawl4ai/stargazers)
[![PyPI version](https://badge.fury.io/py/crawl4ai.svg)](https://badge.fury.io/py/crawl4ai)
[![Downloads](https://static.pepy.tech/badge/crawl4ai/month)](https://pepy.tech/project/crawl4ai)
[![Discord](https://img.shields.io/badge/Discord-join%20us-5865F2?logo=discord&logoColor=white)](https://discord.gg/jP8KfhDhyN)
[![Crawl4AI Cloud](https://img.shields.io/badge/Crawl4AI_Cloud-try_it_free-f5a300?style=flat&labelColor=0d0d10)](https://crawl4ai.com/?ref=readme-badge)

**Latest: [v0.9.4](https://github.com/unclecode/crawl4ai/releases/tag/v0.9.4) (23 Sep 2026)** · [all releases →](https://github.com/unclecode/crawl4ai/releases)

<a href="https://crawl4ai.com/?ref=readme-banner">
  <picture>
    <source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/unclecode/crawl4ai/main/docs/assets/cloud-launch-banner-dark.svg">
    <img alt="Crawl4AI Cloud is live. Soft launch: your first $10 is on us until 31 December 2026, no card. Get your key." src="https://raw.githubusercontent.com/unclecode/crawl4ai/main/docs/assets/cloud-launch-banner-light.svg" width="960">
  </picture>
</a>

</div>

Crawl4AI turns any website into clean, LLM-ready Markdown for RAG, AI agents and data pipelines. Run the open-source web crawler and scraper yourself, free forever, or use it hosted with one key: scrape, search and extract through one API, with MCP for your agent.

## Two ways to use Crawl4AI

### 🐍 Run it yourself: open source, forever

```bash
pip install -U crawl4ai
crawl4ai-setup        # installs the browser, once
```

```python
import asyncio
from crawl4ai import AsyncWebCrawler

async def main():
    async with AsyncWebCrawler() as crawler:
        result = await crawler.arun(url="https://news.ycombinator.com")
        print(result.markdown)

asyncio.run(main())
```

Docker server, CLI and every option: [Installation](#installation) · [docs.crawl4ai.com](https://docs.crawl4ai.com)

### ☁️ Or use the cloud: no browsers, no proxies

1. [![Get a key in 10 seconds](https://img.shields.io/badge/Get_a_key_in_10_seconds-%241_pass%2C_no_signup-f5a300?style=for-the-badge&labelColor=0d0d10)](https://crawl4ai.com/?ref=readme)  
   Verify your email and your first $10 pack is on us (until 31 December 2026, then $5 to start). No card.
2. Get any page as Markdown:

   ```bash
   curl -s https://api.crawl4ai.com/scrape \
     -H "Authorization: Bearer $CRAWL4AI_KEY" \
     -H "Content-Type: application/json" \
     -d '{"url": "https://news.ycombinator.com"}' | jq -r .markdown
   ```

   The same key works for `/search`, `/answer`, `/extract` and many URLs at once (`/scrape/batch`, `/scrape/jobs`). Pay as you go: [live prices](https://crawl4ai.com/docs?ref=readme#pricing).
3. Give it to your AI agent. Claude Code shown; [Codex, Cursor and OpenCode →](https://crawl4ai.com/docs?ref=readme#mcp)

   ```bash
   claude mcp add --transport http crawl4ai https://api.crawl4ai.com/mcp \
     --header "Authorization: Bearer $CRAWL4AI_KEY"
   ```

### Which one?

| | 🐍 Library | 🐳 Your own server | ☁️ Crawl4AI Cloud |
|---|---|---|---|
| **Runs the browsers** | you, in your Python process | you, in Docker on your machine | we do |
| **JS-heavy pages and bot walls** | your settings, your proxies | your settings, your proxies | handled for you, automatically |
| **Web search** | – | – | `/search` and `/answer` |
| **Price** | free, forever | free (your hosting) | pay as you go; your first $10 is on us |

<details>
  <summary>🤓 <strong>My Personal Story</strong></summary>

I grew up on an Amstrad, thanks to my dad, and never stopped building. In grad school I specialized in NLP and built crawlers for research. That’s where I learned how much extraction matters.

In 2023, I needed web-to-Markdown. The “open source” option wanted an account, API token, and $16, and still under-delivered. I went turbo anger mode, built Crawl4AI in days, and it went viral. Now it’s the most-starred crawler on GitHub.

I made it open source for **availability**, anyone can use it without a gate. Now I’m building the platform for **affordability**, anyone can run serious crawls without breaking the bank. If that resonates, join in, send feedback, or just crawl something amazing.

That platform is live now: [Crawl4AI Cloud](https://crawl4ai.com/?ref=readme).
</details>

<details>
  <summary>Why developers pick Crawl4AI</summary>

- **LLM-ready output**: smart Markdown with headings, tables, code and citation hints
- **Fast in practice**: async browser pool, caching, minimal hops
- **Full control**: sessions, proxies, cookies, user scripts, hooks
- **Adaptive intelligence**: learns site patterns, explores only what matters
- **Deploy anywhere**: no keys needed, CLI and Docker, or the hosted cloud
</details>

## ✨ Features

<details>
<summary>📝 <strong>Markdown generation</strong></summary>

- 🧹 **Clean Markdown**: headings, lists, tables and code blocks, in a structure an LLM reads well.
- 🎯 **Fit Markdown**: filters remove menus, footers and boilerplate: `PruningContentFilterLXML`, `BM25ContentFilter` (for a query) and `LLMContentFilter`.
- 🔗 **Citations**: page links become a numbered reference list.
- 🛠️ **Your own strategy**: plug in a custom Markdown generator.

☁️ Same in the cloud: `POST /scrape` returns this Markdown, with no browser to run. [Docs →](https://crawl4ai.com/docs?ref=readme#scrape)
</details>

<details>
<summary>📊 <strong>Structured data extraction</strong></summary>

- 🔎 **CSS and XPath schemas**: fast extraction with no LLM (`JsonCssExtractionStrategy`, `JsonXPathExtractionStrategy`, `RegexExtractionStrategy`).
- 🪄 **Schema generator**: describe what you want once; `generate_schema` writes a reusable schema.
- 🤖 **LLM extraction**: any LLM provider, open-source or hosted, into a typed JSON schema (`LLMExtractionStrategy`).
- 🧱 **Chunking**: topic, regex and sentence chunking for long pages.
- 🌌 **Cosine similarity**: find the chunks that match a query (`CosineStrategy`).

☁️ Same in the cloud: `POST /extract`, with no LLM key of your own. [Docs →](https://crawl4ai.com/docs?ref=readme#extract)
</details>

<details>
<summary>🌐 <strong>Browser control</strong></summary>

- 🖥️ **Your own browser**: persistent profiles with saved logins, cookies and settings.
- 🔄 **Remote browsers**: connect over the Chrome DevTools Protocol (CDP).
- 🔒 **Sessions**: keep a browser state across multi-step crawls.
- 🧩 **Proxies**: with authentication and rotation.
- 🕶️ **Stealth mode**: `enable_stealth`, and an undetected-browser adapter for sites that detect automation.
- ⚙️ **Full control**: headers, cookies, user agents, viewport.
- 🌍 **Chromium, Firefox and WebKit**.
</details>

<details>
<summary>🔎 <strong>Crawling and scraping</strong></summary>

- 🕸️ **Deep crawl**: BFS, DFS and best-first strategies, with crash recovery (`resume_state`) for long crawls.
- 🧠 **Adaptive crawling**: `AdaptiveCrawler` stops when it has learned enough to answer your query.
- 🌱 **URL discovery**: `AsyncUrlSeeder` (sitemaps, Common Crawl) and `DomainMapper`; `prefetch=True` finds URLs 5 to 10 times faster.
- 🚀 **Dynamic pages**: run JavaScript, wait for elements, scroll the full page (`scan_full_page`) for infinite scroll and lazy images.
- 📸 **Screenshots and PDFs** of any page.
- 🖼️ **Media and links**: images, audio, video, `srcset`, internal and external links, iframes, metadata.
- 📂 **Raw HTML and local files**: `raw:` and `file://`.
- 🛠️ **Hooks** at every step of a crawl.
- 💾 **Caching** to skip repeated fetches.
- ⚡ **Many URLs at once**: `arun_many` with a memory-adaptive dispatcher.

☁️ Same in the cloud: up to 50 URLs in one streamed call, or 10,000 in a background job. [Docs →](https://crawl4ai.com/docs?ref=readme#batch)
</details>

<details>
<summary>🐳 <strong>Self-hosting (Docker)</strong></summary>

- 🔐 **Secure by default**: every endpoint needs your `CRAWL4AI_API_TOKEN`.
- 🧰 **REST API**: `/md`, `/html`, `/crawl`, `/crawl/stream`, `/screenshot`, `/pdf`, `/execute_js`.
- 🤖 **MCP**: connect Claude Code and other agents to your own server.
- 📊 **Monitoring dashboard and playground**, a browser pool with pre-warmed pages.
- 🏗️ **AMD64 and ARM64** images.

☁️ Rather not run a server? The cloud is the same idea, hosted. [Get a key →](https://crawl4ai.com/?ref=readme)
</details>

<details>
<summary>☁️ <strong>What the cloud adds</strong></summary>

- 🔍 **Web search API**: `GET /search`, browser-free, ranked and cleaned. [Docs →](https://crawl4ai.com/docs?ref=readme#search)
- 💬 **Answers**: `GET /answer` gives a direct answer to a question (experimental). [Docs →](https://crawl4ai.com/docs?ref=readme#answer)
- 🧪 **Extraction without your own LLM key**: `POST /extract`. [Docs →](https://crawl4ai.com/docs?ref=readme#extract)
- 🧗 **JS-heavy pages and bot walls**: handled automatically; you never pick an engine. [Docs →](https://crawl4ai.com/docs?ref=readme#scrape)
- 🤝 **MCP for your agent**: one line in Claude Code, Codex, Cursor or OpenCode. [Docs →](https://crawl4ai.com/docs?ref=readme#mcp)
</details>

<a id="installation"></a>

## 🛠️ Installation

<details>
<summary>🐍 <strong>pip</strong></summary>

```bash
pip install -U crawl4ai
crawl4ai-setup      # installs and sets up the browser
crawl4ai-doctor     # checks the installation
```

If the browser setup fails, install it by hand:

```bash
python -m playwright install --with-deps chromium
```

Pre-release versions: `pip install crawl4ai --pre`

**Development install**, for contributors:

```bash
git clone https: