pikelet

# Pikelet

**Knowledge that ships as a file.**

Pikelet compiles a corpus into one self-contained, queryable artifact.

A model can interrogate a 456,153-record knowledge base whose backend is a static file.

A `.pikelet` can carry the source text, semantic index, keyword index, query encoder, integrity commitments, retrieval calibration, and evaluation fixtures needed to interrogate that corpus. Put the file on disk, S3, R2, a CDN, or any static HTTP host. A reader can mount it locally or over HTTP Range and search it without a vector database, embedding API, or retrieval server.

Requires Node 20+.

```bash
npx pikelet compile --source ./docs --out docs.pikelet
```

```text
Ingested 3 docs -> 3 chunks
Embedded 3/3 chunks with inline transformer
Built complete .pikelet artifact with 24.5 MB
Compiled docs.pikelet
  24.5 MB, 3 records, identity 8d731a...
```

First run fetches the ~25 MiB query encoder from a GitHub release and caches it; every `.pikelet` file is at least that size regardless of corpus, because the encoder ships inside it — a 5-file folder and a 500-file folder both start around 25 MiB.

Then query it from an LLM — Claude Code, Claude Desktop, or any MCP client:

```bash
npx pikelet mcp install --client claude-code --pack ./docs.pikelet
# or --client claude-desktop; any other MCP client can run
# `npx pikelet mcp --pack ./docs.pikelet` directly, no install step
```

```text
Wrote MCP server "knowledge-packs" to ./.mcp.json
Claude Code picks it up on the next session in this project.
```

Claude now has a `search` tool over your docs. Or query it directly from code:

```js
import { openPikeletFile } from 'pikelet-wasm/complete';
const pack = await openPikeletFile('docs.pikelet');
const out = await pack.query('how do workers restore snapshots', { k: 5 });
console.log(out.matchQuality, out.results[0]?.title);
// 'strong' 'Snapshot restore'
```

That's the whole loop. `compile` also takes a live URL (`--source https://docs.example.com`) instead of a directory. If you want a deployed search app — a Worker + UI, not a file — use `npx pikelet create` instead; see [`pikelet/README.md`](pikelet/README.md) for the full CLI reference and the tradeoffs between the two. `compile` never needs the scaffold path's `@xenova/transformers` dependency; skip its ~140 MB install with `npm install -g pikelet --omit=optional` if you only need `compile`/`mcp`.

Under the hood, the file is one container for everything a reader needs:

```text
documents
    │
    ▼
┌──────────────────┐
│   docs.pikelet   │
│                  │
│ corpus           │
│ semantic index   │
│ lexical index    │
│ query encoder    │
│ integrity        │
│ calibration      │
│ evaluation       │
└──────────────────┘
    │
    ├── local file
    └── static HTTP / object storage
              │
              ▼
        browser / Node / edge / agent
```

Search is the interface. **The file is the knowledge deployment unit.**

---

## The shortest demonstration

A real Pikelet pack contains **456,153 Simple English Wikipedia records** in a **648.5 MiB** artifact.

Try the live browser demo:

```text
https://pikelet-wiki-playground.pages.dev/
```

The page is static HTML/JS on Cloudflare Pages. It mounts the pack from R2:

```text
https://pub-6da2384a3bca4a44b2b2fa29a94cc811.r2.dev/wikipedia.pikelet
```

There is no `/search`, `/embed`, `/query`, or `/api` backend behind the demo.
The browser opens the `.pikelet` file over HTTP Range, verifies the artifact,
loads the embedded query encoder, and shows the byte ranges it reads.

That file was served by a deliberately dumb HTTP server whose only relevant capability was:

```text
fs.createReadStream + HTTP Range
```

There was no vector database, no search service, no embedding service, no application server that understood the pack, no index process on the server.

The pack was mounted by URL and exposed to a real headless Claude process through MCP. Each invocation was restricted with `--strict-mcp-config` to only:

```text
mcp__wiki-pack__search
mcp__wiki-pack__list_packs
mcp__wiki-pack__get_record
```

No web tool. No filesystem tool. No alternate retrieval tool.

One of the actual responses:

> **Q: What causes earthquakes?**
>
> Earthquakes are caused by the movement of tectonic plates in the Earth's crust. Stress builds up along fault boundaries where plates grind against or ride over one another; when that stress exceeds the friction holding the locked fault in place, it suddenly releases as shock waves (seismic waves) that shake the ground [record 429637].
>
> There are three main fault types that can produce quakes — normal, reverse (thrust), and strike-slip — and some quakes have less clear-cut causes, like deep shifts in ancient rock or mine-collapse "tremors" [record 429638].
>
> Two related mechanisms compound this:
>
> * Subduction, where one plate is forced under another into the mantle, produces the most powerful earthquakes and can trigger tsunamis [record 132833].
> * More broadly, the slow convection-driven movement of crustal plates over the hot mantle is the underlying driver of quakes, volcanoes, and mountain formation [record 132267].
>
> Volcanic eruptions can also cause earthquakes, though these are localized to areas near active volcanoes [record 429637].

The model did not just repeat the first search hit. It retrieved multiple records and synthesized them into a sourced answer.

Other runs produced:

```text
Who was the first person on the moon?          → Neil Armstrong          (records 408686, 408684)
How do volcanoes form?                          → plates + subduction + hotspots
                                                                          (records 132770, 132765, 132852, 132853)
How does photosynthesis work?                   → two-phase reaction, cited (5 records: 114219, 114220, 114222, 114223, 269279)
What is the capital of France?                  → Paris                  (records 454641, 302684)
```

That is the product in one demonstration:

> **A model is interrogating a 456,153-record knowledge base whose backend is a static file.**

The model may already contain some of these facts in its pretrained parameters. This test demonstrates the retrieval, synthesis, citation, and deployment path; the Veyra ablation below tests whether support changes when evidence is removed from the pack.

**Network cost.** In a single persistent session (one mount, five queries, one repeat):

| Operation                       | Bytes transferred | Requests |
| ------------------------------- | ----------------: | -------: |
| Initial mount                   |          51.7 MiB |       11 |
| Query 1 + one-time encoder load |          25.9 MiB |       77 |
| Query 2                         |         504.4 KiB |       37 |
| Query 3                         |         499.4 KiB |       46 |
| Query 4                         |         968.9 KiB |       75 |
| Query 5                         |           1.1 MiB |      116 |
| Repeated query 1                |         238.4 KiB |        4 |
| **Total**                       |      **80.8 MiB** |  **366** |

Roughly **12.5% of the 648.5 MiB artifact** crossed the wire across that whole session. Excluding the one-time ~25 MiB encoder load, fresh-query traffic ran **~0.5–1.1 MiB per query**. The complete artifact was never downloaded.

The headless-Claude test above used a fresh process per question, so each invocation repaid the ~52 MiB mount and ~25 MiB encoder cost — about 78 MiB per cold query. That's a real operational distinction: **persistent sessions amortize mount and encoder cost; independent cold processes do not.**

*(The large benchmark fixture still carries its historical `.pancake` filename from before the Pikelet rename; current artifacts use `.pikelet`.)*

---

## What happens if the answer is changed or removed from the file?

This is different from asking whether search returns sensible documents. To test whether Pikelet's retrieval-quality signal could form a useful evidence boundary, a synthetic corpus called **Station Veyra Registry** was built. One version contained the fact:

```text
The Tovash project is housed in Chamber 17.
```

A second pack was byte-for-byte identical except that the record containing that fact was removed.

```text
full pack:     matchQuality: strong    confidence: 0.915   → Chamber 17
ablated pack:  matchQuality: none      confidence: 0.136   → unsupported
```

After the model had already seen the answer, it was queried against the ablated pack with prompts like "Confirm Tovash is in Chamber 17" and "Tovash project Chamber 17 location." The retrieval result stayed unsupported and the model declined to confirm the location from the pack. A separate, informal session then probed a fresh isolated agent with neutral prompts, leading prompts, authority pressure, invitations to use general knowledge, cross-record distractors, and repeated pressure — six adversarial framings — and it continued distinguishing supported facts from the removed one in every case; that session wasn't captured as a script, so treat it as a described observation rather than a reproducible result.

This does **not** mean Pikelet can prevent an LLM from hallucinating. It means the artifact can expose an explicit evidence boundary that a consuming model can choose to respect — and removing evidence from the artifact changed what that model was able to support from the mounted source.

A second intervention changed only the Tovash location source record from **Chamber 17** to **Chamber 43**, rebuilt the pack, and repeated the same prompt in a fresh session:

```text
chamber43 pack: matchQuality: strong    confidence: 0.916   → Chamber 43
```

The model answered **Chamber 43** and cited the same logical source record. With Veyra not mounted at all, the same prompt produced no chamber number and the model declined to guess. Change the evidence and the grounded answer changes with it; remove the evidence source and the a