browser-scraper

# Browser Scraper: the stealth browser for web scraping and AI agents

**Undetected Chrome for Node.js that reads and operates the sites that block bots. Use it as a library, a CLI or an MCP server for Claude, Cursor and Codex. It turns any page, including closed Shadow DOM and cross-origin iframes, into clean Markdown, structured data or an accessibility snapshot an agent can click through. Local, free, MIT, bring-your-own-LLM.**

[![npm version](https://img.shields.io/npm/v/@eurafaeldecarvalho/browser-scraper.svg)](https://www.npmjs.com/package/@eurafaeldecarvalho/browser-scraper)
[![CI](https://github.com/eurafaeldecarvalho/browser-scraper/actions/workflows/ci.yml/badge.svg)](https://github.com/eurafaeldecarvalho/browser-scraper/actions/workflows/ci.yml)
[![npm downloads](https://img.shields.io/npm/dm/@eurafaeldecarvalho/browser-scraper.svg)](https://www.npmjs.com/package/@eurafaeldecarvalho/browser-scraper)
[![node](https://img.shields.io/node/v/@eurafaeldecarvalho/browser-scraper.svg)](https://nodejs.org)
[![license: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](./LICENSE)

```ts
import { scrape } from "@eurafaeldecarvalho/browser-scraper";

const page = await scrape({ url: "https://example.com", formats: ["markdown", "metadata"] });
console.log(page.tier);     // "http" | "browser" | "stealth": the cheapest tier that worked
console.log(page.markdown); // clean, LLM-ready Markdown of the main content
```

```bash
# Give Claude Code a browser that gets through Cloudflare:
claude mcp add browser-scraper -- npx -y @eurafaeldecarvalho/browser-scraper mcp
```

`@eurafaeldecarvalho/browser-scraper` drives real Chrome over the raw DevTools Protocol (CDP). It never sends `Runtime.enable`, one of the most widely fingerprinted automation signals, and every layer of the browser's identity tells the same story. On top of that stealth core it adds the pieces AI workloads need: Markdown and metadata extraction, an accessibility snapshot with clickable refs, a one-call `scrape()` that escalates from plain HTTP to a stealth browser only when it has to, an MCP server, and optional LLM extraction with any model.

**Why this one:**

- **Gets through where headless browsers get blocked.** There is no `Runtime.enable` leak, the persona (UA, Client Hints, WebGL, timezone, locale) is coherent, and mouse and keyboard input are human-like. `solveTurnstile()` handles Cloudflare's checkbox.
- **Sees everything.** `find()`, `markdown()` and `snapshot()` all reach into **closed** shadow roots and **cross-origin iframes**, which Playwright's snapshot and Readability-style extractors miss.
- **Built for agents.** The MCP server and CLI work with no extra install. Credentials are typed through `{{SECRET}}` placeholders so the model never sees them, an allow-list restricts domains, and page content is fenced as untrusted data.
- **LLM only where it pays.** `generateExtractor()` has the model write CSS selectors **once**. After that `run()` extracts with no LLM, and `runOrHeal()` regenerates the selectors when the site changes.
- **Local and free.** No cloud, no API key, no per-page billing. You bring the proxy and, optionally, the model (OpenAI, Anthropic, Google, Ollama… through the Vercel AI SDK, or your own function).

## Passes the bot detectors people actually use

With the right setup — a clean residential/mobile IP, plus a real GPU (EC2 `g4dn`/`g5`) for render-hash targets — `browser-scraper` is built to clear the fingerprint suites and anti-bot stacks that block headless automation:

| Detection / fingerprint suite | | Anti-bot / WAF vendor | |
| --- | :---: | --- | :---: |
| `bot.sannysoft.com` — 0 failures | ✅ | Cloudflare (Turnstile) | ✅ |
| CreepJS — 0% headless, 0% stealth | ✅ | DataDome | ✅ |
| `bot-detector.rebrowser.net` | ✅ | Kasada | ✅ |
| Pixelscan | ✅ | Akamai Bot Manager | ✅ |
| IPHey | ✅ | PerimeterX / HUMAN | ✅ |
| Browserscan | ✅ | Imperva (Incapsula) | ✅ |
| Incolumitas | ✅ | Shape / F5 | ✅ |
| Fingerprint.com | ✅ | AWS WAF Bot Control | ✅ |

> **A ✅ means the fingerprint + behavior layer passes — it is not a guaranteed bypass.** Against hard targets your **IP reputation is the ceiling**: a datacenter ASN loses before the JavaScript runs, so route through a residential/mobile proxy (see [The production stack](#the-production-stack)). `browser-scraper` is **not** a CAPTCHA solver — for reCAPTCHA v2 / hCaptcha / Arkose (FunCaptcha) image challenges, pair it with a solver service. It *does* keep reCAPTCHA **v3** scores high via human-like input and warmed profiles.

> Numbers were taken on a desktop Chrome. On a GPU-less Lambda running **chrome-headless-shell** (`@sparticuz/chromium`) the surfaces differ — re-run the suites against your *actual* production binary and treat that as the source of truth (see [Cloud deployment](#cloud-deployment-lambda-vs-ec2-gpu)).

This is a precision tool, not a magic bypass — it's honest about its edges (see [Notes & limitations](#notes--limitations)), and that honesty is exactly what makes the green checks above worth anything.

---

## Contents

- [Install](#install)
- [Quickstart](#quickstart)
- [For AI agents & LLMs](#for-ai-agents--llms) — Markdown, `scrape()`, snapshot + refs, MCP server, CLI, LLM extraction
- [The tour](#the-tour) — what it does, by example
  - [Pierce closed Shadow DOM + cross-origin iframes](#pierce-closed-shadow-dom-and-cross-origin-iframes-with-one-find)
  - [Wait for the right moment](#wait-for-the-right-moment)
  - [Interact like a human](#interact-like-a-human)
  - [Control the network](#control-the-network)
  - [Coherent browser fingerprint](#coherent-browser-fingerprint-user-agent-client-hints-webgl-timezone)
  - [Persistent profiles & warming](#persistent-profiles--warming)
- [API reference](#api-reference)
- [Stealth, in depth](#stealth-in-depth)
- [WebGL / GPU on cloud hosts](#webgl--gpu-on-cloud-hosts)
- [The production stack](#the-production-stack) — proxy · VPS · solver
- [Cloud deployment (Lambda vs EC2-GPU)](#cloud-deployment-lambda-vs-ec2-gpu)
- [Identity & persistence](#identity--persistence)
- [WebRTC](#webrtc)
- [Alternatives — how it compares](#alternatives--how-it-compares)
- [FAQ](#faq)
- [Notes & limitations](#notes--limitations)
- [Support](#support)

---

## Install

```bash
npm install @eurafaeldecarvalho/browser-scraper
```

or:

```bash
pnpm add @eurafaeldecarvalho/browser-scraper
```

Requires Node.js ≥ 20 and Chrome/Chromium. Profile history seeding uses Node's built-in `node:sqlite` (Node ≥ 22.5) or the optional `better-sqlite3`; if neither is available it is simply skipped. The LLM helpers need `ai` (and `zod` if you use Zod schemas), and only if you import `/ai`.

Requires Node.js ≥ 20 and a local Chrome/Chromium (it's found automatically, or point at one with `chromePath`). For AWS Lambda, also install the optional `@sparticuz/chromium` — see [Cloud deployment](#cloud-deployment-lambda-vs-ec2-gpu).

## Quickstart

```ts
import { Browser } from "@eurafaeldecarvalho/browser-scraper";

const browser = new Browser();

try {
  const tab = await browser.newTab();
  await tab.goto({ url: "https://example.com" });

  const heading = await tab.find({ selector: "h1" });
  console.log(await heading?.text());

  await tab.screenshot({ path: "example.png" });
} finally {
  await browser.close();
}
```

That's the whole shape of the library: a `Browser`, a `Tab`, and `Element`s you find and act on. Everything below is detail on top of these three.

---

## For AI agents & LLMs

### Clean Markdown, metadata and links

```ts
const tab = await browser.newTab();
await tab.goto({ url: "https://shop.example.com/product/42" });

const { markdown, title, links, wordCount } = await tab.markdown({
  mainContent: true,      // drop nav/header/footer/cookie banners (default)
  query: "price shipping", // optional: keep only the relevant blocks (BM25), saves tokens
  links: "inline",        // "inline" | "reference" | "none"
  includeIframes: false,  // true = also inline iframe documents, cross-origin included
});

const meta = await tab.metadata();
// { title, description, canonical, lang, image, openGraph, twitter, jsonLd: [...] }
// JSON-LD often already has the price, rating and availability: check it before you prompt.
```

`markdown()` reads Chrome's own DOM tree over CDP (`DOM.getDocument` with `pierce`). **No script is injected into the page**, and the content of **closed shadow roots** is included in reading order, with slots composed. For HTML you already have, `htmlToMarkdown(html)` and `extractMetadata(html)` do the same without a browser.

### `scrape()`: one call, cheapest tier first

```ts
import { Browser, scrape } from "@eurafaeldecarvalho/browser-scraper";

const browser = new Browser({ proxy: process.env.PROXY });   // reuse it across URLs
const page = await scrape({
  url,
  browser,
  formats: ["markdown", "metadata", "links", "html", "screenshot"],
  strategy: "auto",   // http → browser → stealth (or pin one tier)
  http: { curlImpersonate: "curl_chrome131" }, // optional: real Chrome TLS for the http tier
});

page.tier;      // which tier produced the result
page.attempts;  // [{ tier: "http", ok: false, reason: "needs JavaScript (empty app shell)", ms: 4 }, …]
```

| Tier | What runs | Used when |
|---|---|---|
| `http` | One request, no Chrome (Node fetch, or curl-impersonate for a genuine Chrome JA3/JA4) | Static pages, APIs |
| `browser` | The stealth Chrome | The HTTP response was blocked (403/429/503), a challenge, or an empty JS app shell |
| `stealth` | The same tab plus a behavioral warm-up and an interactive Turnstile solve | The browser still sees a challenge page |

A challenge page is **never** returned as content: if every tier is blocked, `scrape()` throws a `ScrapeError` that carries the `attempts`.

### Accessibility snapshot + refs (what agents click on)

```ts
console.log(await tab.snapshot());
// - main
//   - heading "Sign in" [level=1]
//   - textbox "Email" [ref=e3]
//   - button "Continue"