webcrawler-deep-crawl

# Website Deep Crawl

> Input: one or more start URLs (+ optional scope, depth, page-count, globs, removal selectors). Output: per-page records `{url, crawl, metadata, text, markdown, html, outboundLinks}` for every page reached within scope.

## Language

All process output to user (progress updates, process notifications) follows the user's language.

## Objective

From a small set of start URLs, breadth-first crawl every reachable in-scope page, strip boilerplate (navigation, header, footer, cookie banners, etc.), and emit per-page LLM-ready content (text / markdown / HTML) plus structured metadata — suitable for feeding RAG pipelines, vector databases, or chatbot knowledge bases.

## Prerequisites

- One or more start URLs are provided by the caller.
- Target pages are publicly reachable, OR the running browser is already logged in for any pages behind authentication.
- A working directory is available for writing per-page JSON records and the crawl state file.

## Pre-execution Checks

### 1. Tool Readiness

If browser-act has been confirmed available in the current session → skip this step.

Invoke `browser-act` via Skill tool to load usage. If installation or configuration issues arise, follow its guidance to resolve then retry.

### 2. Login Verification (when prerequisites include login requirement)

If login status for the target site has been confirmed in the current session → skip this step.

Otherwise: open the target site and observe the page login status:
- Logout/sign-out entry, user avatar, or username exists → logged in, continue execution
- Login/register entry exists with no logout entry → not logged in, inform the user that login is needed first, assist the user in completing the login flow

User refuses or cannot log in → terminate execution.

## Capability Components

> This Skill's operational boundary = what the user can manually do in their browser. It only reads data already displayed to the user on the page, never bypassing authentication or access controls. Its role is equivalent to copy-pasting on the user's behalf — the data is already on screen, automation merely saves time. JS code is encapsulated in Python files under the `scripts/` directory, invoked via `browser-act --session <name> eval "$(python scripts/xxx.py {params})"`. `$(...)` is bash syntax; use the bash tool for execution. The `eval` token below refers to the browser-act CLI `eval` subcommand — always include the `browser-act --session <name>` prefix and the `"$(...)"` substitution.

Below are all atomic capabilities discovered and verified during the exploration phase, listed by command template with parameters. Simply invoke them as needed — no need to read `scripts/*.py` source code or re-verify. Only inspect scripts when execution fails for troubleshooting. Combine freely as needed during execution.

### API: discover URLs from /llms.txt

Probes `{origin}/llms.txt` (a convention used by LLM-friendly documentation sites) and returns every URL it lists. Fast path — try this first.

`eval "$(python scripts/discover-llms-txt.py 'https://example.com')"`

Parameters:
- positional `origin`: the site origin (scheme + host), e.g. `https://docs.example.com`

Output example:
```json
{
  "error": false,
  "source": "llms.txt",
  "count": 88,
  "urls": ["https://docs.example.com/intro", "https://docs.example.com/install"]
}
```

On failure (file missing or HTTP error): `{"error": true, "message": "llms.txt not available (HTTP 404)", "urls": []}` — move on to sitemap discovery.

### API: discover URLs from /sitemap.xml

Probes `{origin}/sitemap.xml` and `{origin}/sitemap_index.xml`, follows nested sitemap indexes, and collects every `<loc>` URL.

`eval "$(python scripts/discover-sitemap.py 'https://example.com' --max-urls 5000)"`

Parameters:
- positional `origin`: site origin
- `--max-urls`: hard cap to stop ballooning sitemap indexes, default `5000`

Output example:
```json
{
  "error": false,
  "source": "sitemap.xml",
  "count": 86,
  "urls": ["https://example.com/page-a", "https://example.com/page-b"]
}
```

On failure: `{"error": true, "message": "No sitemap found at standard paths", "urls": []}` — fall back to DOM link discovery.

### DOM: discover URLs from the current page

Reads `<a href>` from the currently loaded page DOM, normalizes them to absolute URLs, drops fragments, asset extensions, and out-of-scope links, and optionally applies include / exclude glob filters. Use when llms.txt and sitemap are both unavailable, or to extend the queue with links discovered while crawling.

`eval "$(python scripts/discover-links.py 'https://example.com/docs/' --include-globs '[]' --exclude-globs '["**/changelog/**"]')"`

Parameters:
- positional `start_url`: scope URL. Only links under its directory (or equal to it) are kept.
- `--include-globs`: JSON array of glob patterns; if non-empty, a link must match at least one to be kept. Default `[]` (no include filter).
- `--exclude-globs`: JSON array of glob patterns; matching links are dropped. Default `[]`.

Glob semantics: `**` matches any characters (including `/`), `*` matches any except `/`, `?` matches one character. Example: `https://example.com/{docs,api}/**`.

Output example:
```json
{
  "error": false,
  "source": "dom",
  "page": "https://example.com/docs/intro",
  "scope_base": "https://example.com/docs/",
  "count": 12,
  "links": ["https://example.com/docs/intro", "https://example.com/docs/install"]
}
```

### DOM: extract clean content + metadata + outbound links from the current page

The core extractor. Run this on every crawled page after `wait stable`. Returns the page body in the requested format(s), structured metadata, and the in-scope outbound links found on this page (so callers can extend the BFS queue without a second DOM pass).

`eval "$(python scripts/extract-page-content.py 'https://example.com/docs/' --output-format markdown --remove-selectors '.cookie-banner,#chat-widget' --include-globs '[]' --exclude-globs '[]')"`

Paramet