dlbrowser

<div align="center">

# DLBrowser

### Your AI agent is blind to the half of the web that fights back.

Cloudflare, captchas, and JavaScript walls block the tools agents use to browse — silently.
**DLBrowser is the one that gets through.** One line. Self-healing. Metered to the credit.

[![PyPI](https://img.shields.io/pypi/v/dlbrowser)](https://pypi.org/project/dlbrowser/)
[![npm](https://img.shields.io/npm/v/dlbrowser)](https://www.npmjs.com/package/dlbrowser)
[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](./LICENSE)
[![Python](https://img.shields.io/pypi/pyversions/dlbrowser)](https://pypi.org/project/dlbrowser/)

```bash
npx dlbrowser connect --key dlb_xxx
```

*Wires DLBrowser into Claude Code, Cursor, Windsurf, Cline & Codex in one command.
Your agent gets 11 web tools that actually reach the page.*

</div>

---

## The 60-second try

```bash
pip install dlbrowser
dlbrowser fetch https://example.com
```

Every fetch tells you **which backend got through and how long it took** — because the
engine picks the cheapest one that works and only escalates when a site fights back:

```json
{ "success": true, "backend": "curl_cffi", "load_time": 0.03, "blocked": false }
```

And every run is metered against a plan, in the open — no surprise invoice:

```json
{ "plan": "scout", "remaining_daily": 497, "remaining_monthly": 2497,
  "upgrade_url": "https://dekryptlabs.github.io/dlbrowser/pricing.html" }
```

That's a real response from the live gateway. Start free — 2,500 credits, no card.

---

## Why your agent needs this

You wire web access into an agent. It works in the demo. Then it meets the real web:

- One site loads with a 50ms HTTP request. The next is an empty React shell until JS runs.
- Cloudflare and DataDome quietly serve your headless browser a challenge page.
- Your agent's built-in fetch returns `""` — not an error, an **empty string** — and keeps going.
- You have no idea which sites failed, which backend would've worked, or what any of it cost.

DLBrowser is the capability layer that fixes this. Not a scraper. Not a crawler.
**Acquisition infrastructure for AI agents** — reliable, observable, billable.

---

## The engine: start cheap, escalate only when blocked

```
curl_cffi  →  tls_client  →  playwright  →  nodriver (CDP)  →  firecrawl
  ~50ms         ~200ms        full JS        max stealth       last resort
  HTTP-level, no browser       │            browser-level, real Chromium
                               └── auto-escalates when a block is detected
```

~70% of real fetches never need a browser. Starting with a 50ms request and climbing the
ladder only on a detected block keeps you fast and cheap — while still getting through the
sites that stop everything else. **Five backends.** Firecrawl has two. Crawl4AI has one.

---

## What resets your expectations

| | |
|---|---|
| 🔁 **Self-healing fallback** | Five backends auto-escalate on block. Your agent stops getting an empty string. |
| 💸 **Per-run cost telemetry** | Every run logs backend, credits, and estimated cost to a queryable ledger. **No competitor exposes this.** Your CFO can finally answer "what does our web data cost?" |
| 🪙 **Token-efficient** | Returns clean markdown/extracted content, not raw HTML — **~4× fewer tokens** into your agent's context on content pages ([measured](#fewer-tokens-cleaner-signal)). Lower API cost, more window, better signal. |
| 🎯 **No JS multipliers** | 1 credit per page on *any* backend. Firecrawl & ScrapingBee charge 5–7× for JS rendering. |
| 🔒 **Atomic metering** | Reservation-based — **0% overshoot**, even under concurrency. You're never billed past your quota. |
| 🧠 **LLM extraction** | Natural language → structured JSON. Runs against a local/free model — no cloud key required. |
| 📦 **MIT, self-hostable** | Full feature parity self-hosted. Your data never leaves your infra. The hosted gateway just saves you running proxies + captcha infra. |
| 🔌 **MCP-native** | 11 tools with use-first descriptions, so your agent reaches for DLBrowser over its flaky built-in browser. |

---

## Fewer tokens, cleaner signal

Raw HTML pours tags, scripts, and boilerplate into your agent's context — you pay tokens
for noise, and the model burns attention parsing `<div class>` soup. DLBrowser returns
clean **markdown** or extracted content instead.

Real measurement through the live gateway — a Wikipedia article:

| Mode | Tokens (approx) | vs raw HTML |
|------|-----------------|-------------|
| Raw HTML | ~57,000 | — |
| **Markdown** | **~13,000** | **~77% fewer** |
| Article | ~42,000 | ~27% fewer |

**~4× less context per fetch on a content page** — lower API cost, more room in the
window, and better signal for the model to reason over. Savings vary by page (a sparse
link list saves little, and your agent picks the mode) — so measure your own:

```python
import asyncio, tiktoken
from hermes_browser import fetch          # pip install dlbrowser
enc = tiktoken.get_encoding("cl100k_base")
raw = asyncio.run(fetch(url, mode="html"))["content"]["html"]
md  = asyncio.run(fetch(url, mode="markdown"))["content"]["markdown"]
print(len(enc.encode(raw)), "→", len(enc.encode(md)), "tokens")
```

## One line, every host

```bash
npx dlbrowser connect --key dlb_xxx
# detects and configures: claude-code · claude-desktop · cursor · windsurf · cline · codex
```

Or point any MCP client at the **free hosted demo** gateway (try it with no local setup —
some features are gated on the demo; self-host for the full engine):

```bash
claude mcp add dlbrowser --transport http \
  --url https://api.dekryptlabs.com \
  --header "Authorization: Bearer dlb_xxx"
```

Your agent immediately has: `fetch` · `scrape` · `smart_extract` · `screenshot` ·
`markdown` · `crawl` · `search` · `serp` · `status` · `account` · `topup`.

---

## Can I trust it? (architecture)

```
Agent / CLI / Extension / MCP client
        │
   ┌────┴─────┐
   │ Gateway  │  Paywalled — 401 no key · 402 over quota · free: status/account
   └────┬─────┘
   ┌────┴─────┐
   │  Router  │  Adaptive per-domain scoring: success · latency · cost · freshness
   └────┬─────┘
   ┌────┴─────────────────────────────┐
   │  curl_cffi → tls → playwright →  │  self-healing cascade
   │  nodriver → firecrawl            │
   └────┬─────────────────────────────┘
   ┌────┴─────┐
   │  Ledger  │  per-run backend · credits · cost · trace_id (OpenTelemetry)
   └──────────┘
```

The router learns per-domain — more usage, better routing. Everything above the gateway is
**MIT and self-hostable**. Read the code before you trust it; that's the point of open source.

---

## Can it beat Cloudflare?

On many real sites, **with no browser at all.** We tested 5 Cloudflare-protected sites from
a datacenter IP with no residential proxy (worst case):

| Site | Result | How |
|------|--------|-----|
| nowsecure.nl (CF challenge test) | ✅ pass | TLS impersonation |
| crunchbase.com | ✅ pass | curl_cffi, 0.3s, no browser |
| nike.com | ✅ pass | curl_cffi, 1.2s, no browser |
| g2.com | ❌ 403 | Cloudflare Enterprise — datacenter IP blocked by reputation |
| zillow.com | ❌ 403 | Enterprise + captcha — needs residential IP |

**3/5 passed on HTTP-level TLS impersonation alone.** The two failures weren't fingerprint
failures — they're IP-reputation blocks that need residential-proxy routing (supported,
disabled in this test). No tool beats Cloudflare universally; DLBrowser's answer is layered
escalation. Full methodology + "what beating Cloudflare actually means" →
[marketing/cloudflare.md](./marketing/cloudflare.md).

We have **not** published broad head-to-head benchmarks vs Firecrawl/Crawl4AI yet — those
are next, with the methodology published alongside. We don't print numbers we made up.
**Don't trust our table — run it on your own hardest sites:**

```bash
pip install "dlbrowser[full]" && python3 -m playwright install chromium
dlbrowser fetch <a-url-that-blocks-your-current-tool>   # see the backend that got through
```

---

## Pricing

| Plan | Price | Credits | Daily cap | Backends | Captcha | Proxy | LLM | Keys |
|------|-------|---------|-----------|----------|---------|-------|-----|------|
| **Scout** | Free | 2,500 one-time | 500 | curl_cffi | — | — | — | 1 |
| **Analyst** | $29/mo | 100,000/mo | 5,000 | + tls_client | — | — | 50k | 3 |
| **Operator** | $85/mo | 500,000/mo | 25,000 | + playwright | ✓ | — | 200k | 10 |
| **Command** | $299/mo | 2,000,000/mo | 100,000 | all + nodriver | ✓ | ✓ | 1M | 50 |
| **Enterprise** | from $999/mo | 10M–100M+/mo | — | all + dedicated | ✓ | ✓ | ∞ | ∞ |

1 credit = 1 page, any backend. Annual 20% off. Self-hosting is free forever.
Full pricing → [PRICING.md](./PRICING.md) · hosted: [dekryptlabs.github.io/dlbrowser/pricing.html](https://dekryptlabs.github.io/dlbrowser/pricing.html)

---

## Will my company approve it?

- **Multi-tenant** — organizations, teams, seats, roles
- **API keys** — create, revoke, rotate, scope, expire (SHA-256 hashed)
- **Audit logs** — immutable event ledger: who did what, when, at what cost
- **Webhooks** — HMAC-SHA256 signed, with a dead-letter queue
- **SLA monitoring** — scheduled page diffing with noise-tolerant change detection
- **Self-hosted** — Docker one-liner, full feature parity, data stays in your network

Enterprise plans publish real prices (from $999/mo) — no "call us" black box. `sales@dekryptlabs.com`

---

## When *not* to use DLBrowser

- You need **pre-built datasets** → Bright Data / Apify
- You need a **visual scraping IDE** → Octoparse
- You need **screenshot-as-a-service** → Urlbox

DLBrowser is for teams running AI agents that need reliable, observable, cost-controlled
web access at scale.

---

## Docs

| Resource | Link |
|----------|------|
| Getting Started | [GETTING_STARTED.md](./GETTING_STARTED.md) |
| Integrations (LangChain / LlamaIndex) | [INTEGRATION_QUICKSTART.md](./INTEGRATION_QUICKSTART.md) |
| Self-Hosted Deploy | [DEPLOYMENT.md](./DEPLOYMENT.md) |
| Comparison vs Firec