docpull

<p align="center">
  <img
    src="https://raw.githubusercontent.com/raintree-technology/docpull/main/docs/launch-assets/logo-square-light-400.png"
    alt="DocPull"
    width="128"
  />
</p>

# docpull

**Context dependencies for AI agents. Browser-free by default.**

[![Python 3.10+](https://img.shields.io/badge/python-3.10+-blue.svg)](https://www.python.org/downloads/)
[![PyPI version](https://img.shields.io/pypi/v/docpull.svg?label=package)](https://pypi.org/project/docpull/)
[![PyPI downloads](https://img.shields.io/pepy/dt/docpull?label=downloads)](https://pepy.tech/project/docpull)
[![GitHub stars](https://img.shields.io/github/stars/raintree-technology/docpull?style=social)](https://github.com/raintree-technology/docpull/stargazers)
[![License: MIT](https://img.shields.io/github/license/raintree-technology/docpull)](https://github.com/raintree-technology/docpull/blob/main/LICENSE)

<!-- mcp-name: io.github.raintree-technology/docpull -->

DocPull is a local-first dependency manager for AI context. Define the public
web sources an agent depends on, sync them into cited context packs, diff what
changed, and export reproducible context for Cursor, Claude, OpenAI,
LlamaIndex, LangChain, MCP clients, and RAG pipelines.

Architecturally, DocPull is the evidence and acquisition engine: it acquires,
versions, cites, hashes, and replays evidence. Downstream products own
scheduling, review, approved claims, legal conclusions, and notifications. See
the [architecture decision](docs/adr/0001-evidence-acquisition-engine.md).

The core workflow is a `docpull.yaml` plus a `.docpull/context.lock.json`,
similar in spirit to code dependency manifests and lockfiles:

```bash
docpull init my-agent-context
docpull add stripe react postgres
docpull install
docpull deps
docpull sync
docpull diff
docpull export context-pack --target openai
```

Bundled aliases such as `stripe`, `react`, `postgres`, `openai`, and
`apple-hig` expand to normal HTTPS sources in `docpull.yaml`. Runs stay
reproducible through the lockfile: source URLs, discovered URLs, content hashes,
run IDs, aliases, and export metadata are recorded without storing secrets.
Use `docpull sources list` to inspect the bundled alias catalog and
`docpull install` to validate or recreate the local dependency lock.
Use `docpull deps` to see the current dependency, lockfile, latest run, and
export status.

Projects can also track typed known-source specs such as `pypi:requests`,
`rfc:9110`, `wiki:Web_scraping`, or a local dataset path. Those sources sync
through their typed lanes and do not use discovery.

The original `docpull URL ...` workflow still works: fetch public or explicitly
authorized static/server-rendered web pages and write clean Markdown, NDJSON,
SQLite, or Open Knowledge Format (OKF) outputs. OKF is a portable bundle format
that keeps content, indexes, and manifests together for reuse across tools.
Project mode adds the persistent evidence lifecycle on top: sources, runs,
diffs, exports, evals, accounting, and local auditability.

DocPull is local-first: direct fetching, sitemap/link discovery, extraction,
indexing, pack intelligence, and opt-in `agent-browser` rendering can run with
no external account and no required API spend. Cloud rendering is explicit and
budget-guarded.

DocPull aligns core workflows across CLI, Python SDK, and MCP, with each surface
optimized for its user. The [Surface Contract](docs/surface-contract.md) defines
how those surfaces align and where they intentionally differ.
For the context dependency workflow, see
[Context Dependencies](docs/context-dependencies.md).

Web-source ingestion is the core workflow. Documentation is one high-value
lane, not the product boundary. It works best on static or server-rendered
pages such as blogs, API references, OpenAPI specs, changelogs, vendor pages,
product pages, filings, docs sites, and other pages where the useful content is
available in HTML or embedded page data.

Direct fetches also recognize standards-style plain text and common textual
formats from their response media type, including extensionless URLs. Remote
documents remain denied by default; explicitly opt into local PDF parsing with
`docpull URL --remote-documents pdf` after installing `docpull[markitdown]` or
`docpull[parse]`. This does not enable a browser, cloud parser, or paid route.

DocPull is browser-free by default. JS-only pages are skipped with a clear
reason unless you explicitly opt into a local renderer. See
[Web Source Boundary](docs/scraping-boundary.md) and
[Alternatives](docs/alternatives.md) for the full boundary.

## Install

```bash
pip install docpull
```

## Project Quickstart

```bash
docpull init stripe-docs
docpull add stripe
docpull install
docpull sync
docpull deps
docpull diff
docpull export context-pack --target cursor
```

## Context CI

Use Context CI when an agent loop depends on current, cited context and a
missing or stale source should fail the build:

```bash
docpull ci --prepare
```

`docpull ci` runs locally against either a project root or a standalone pack. It
checks the project lockfile, pack score, pack audit, coverage confidence,
citation coverage, eval-grade sidecars, evidence basis quality, rights
metadata, and optional context predictions. It writes `context-ci.report.json` and `CONTEXT_CI.md`;
the command exits non-zero when hard gates fail.

Minimal GitHub Actions job:

```yaml
name: Context CI
on: [pull_request]
jobs:
  context:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.11"
      - run: pip install docpull
      - run: docpull ci --prepare
```

For the full workflow, see [Context CI](docs/context-ci.md). The durable
artifact shape is documented in
[Context Pack Contract v3](docs/context-pack-contract-v3.md).

Example diff after a later sync:

```text
Project diff: +4 -2 ~18 api=2 pricing=1

Changed pages:
- /payments/payment-intents
  likely API behavior change
- /billing/subscriptions
  pricing / billing change
- /webhooks
  likely API behavior change

0 failed URLs
0 robots blocked
0 paid/cloud routes used
```

## Context Pack Contract

DocPull writes three explicit layers of artifacts:

| Layer | Purpose | Contract check |
| --- | --- | --- |
| Raw extraction | Fetched documents, chunks, routes, and source index sidecars | `docpull pack validate PACK --level raw` |
| Agent-ready pack | Raw evidence plus citation index, coverage, score, audit, and lock sidecars | `docpull pack validate PACK --level agent` |
| Eval-grade pack | Agent pack plus rights, provenance, basis/eval artifacts, and pack card | `docpull pack validate PACK --level eval` |

Core ingestion paths write into the same v3 contract:

```bash
docpull https://docs.example.com -o packs/docs
docpull parse ./handbook.pdf -o packs/handbook --backend auto
docpull openapi-pack ./openapi.json -o packs/api
docpull feed-pack https://example.com/news -o packs/news
docpull paper-pack arxiv:1706.03762 -o packs/papers
docpull repo-pack psf/requests -o packs/repo --cache
docpull package-pack pypi:requests -o packs/package
docpull standards-pack rfc:9110 -o packs/standard
docpull dataset-pack ./metrics.csv -o packs/dataset
docpull dataset-pack 'https://data.example.org/rows.json?$limit=100' -o packs/dataset
docpull relationship-pack example.com --subject "Example Brand" -o packs/relationships
docpull website-pack example.com -o packs/website
docpull transcript-pack ./meeting.vtt -o packs/transcript
docpull wiki-pack wiki:Web_scraping -o packs/wiki
docpull brand-pack example.com -o packs/brand
docpull product-pack https://example.com/pricing -o packs/product
docpull styleguide-pack example.com -o packs/styleguide
docpull image-pack example.com -o packs/visuals
docpull policy-pack example.com -o packs/policies
docpull pack prepare packs/docs --eval-grade
docpull pack validate packs/docs --level eval
docpull export packs/docs --format openai-vector-jsonl -o exports/openai.jsonl
docpull export packs/docs --format cursor-rules -o .cursor/rules --skill-name docs
```

The canonical baseline-aware website acquisition contract is documented in
[Website snapshots](docs/website-snapshot.md).

Use `docpull ci --prepare` to validate a project or standalone pack in CI.

Install optional extras as needed:

```bash
pip install 'docpull[llm]'           # tiktoken for token-accurate chunking
pip install 'docpull[trafilatura]'   # alternative extractor for noisy pages
pip install 'docpull[parse]'         # MarkItDown + Unstructured local document parsers
pip install 'docpull[presidio]'      # optional Presidio PII detection for redaction
pip install 'docpull[mcp]'           # stdio MCP server
pip install 'docpull[serve]'         # local pack JSON server runner
pip install 'docpull[parquet]'       # optional Parquet export support
pip install 'docpull[e2b]'           # E2B cloud sandbox renderer SDK
```

Prefer installing the extras needed for the current lane instead of a broad
bundle. The base install remains useful without API keys or paid services.

Browser rendering is an explicit external extension, not part of the base
install. Install an `agent-browser` compatible CLI separately, put it on
`PATH`, or set `DOCPULL_AGENT_BROWSER_BIN=/path/to/agent-browser`. Verify the
runtime with `docpull render --check`. Render targets must use HTTPS except for
localhost/loopback HTTP during local testing, and DocPull keeps renderer action
permissions locked down to HTML retrieval only. Because browser rendering
cannot fully enforce redirect, subresource, or connect-time DNS allow-lists,
network browser rendering fails closed unless the operator sets
`DOCPULL_RENDER_TRUSTED_BROWSER_TARGETS=1` for trusted targets. For
localhost/loopback HTTP tests, set `DOCPULL_RENDER_ALLOW_LOCAL_TARGETS=1`.

For stronger isolation, cloud runtimes are available explicitly:
`docpull render URL --runtime vercel` uses the Vercel Sandbox CLI and Vercel
auth, while `doc