memory-corpus-ingest

# Corpus Ingest

Bring a large dataset into the assistant's working knowledge without stuffing it into memory. The model is a library: the workspace holds the stacks (the raw files, cold and complete), memory holds the card catalog (a small set of map pages that say what exists, when it is from, and where to look), and a purpose-built retrieval skill is the librarian that walks to the right shelf on demand.

Two invariants drive everything below:

1. **Raw data never enters the memory corpus.** Nothing from the dataset is written into `memory/concepts/` except the map pages, and nothing is ever appended to `memory/buffer.md` (bulk buffer appends trip the consolidation burst guard and per-run caps; the map bypasses the buffer entirely via `assistant memory ingest`).
2. **The map stays small.** Roughly 10 to 50 pages regardless of corpus size. If the corpus doubles, the pages get denser or the slices get coarser; the page count does not double.

## Procedure

### Step 1: Scope and confirm

Identify the source and its size before committing:

```bash
du -sh /path/to/raw-corpus
find /path/to/raw-corpus -type f | wc -l
```

Tell the user what will happen: the raw files move into the workspace, a bounded number of summarization passes read them once to build the map, the map is ingested into memory, and a lookup skill is authored for drill-in. Skimming a large corpus is real LLM work that costs time and money; confirm before starting. For Fathom recording exports, read `references/fathom.md` first for format discovery and slicing guidance.

### Step 2: Cold-store the raw corpus

Land the raw files under an imports directory in the workspace, one directory per source:

Screen for credentials BEFORE copying: an arbitrary corpus can carry secret
material, and anything landed under `imports/` becomes reachable by workspace
tools, backups, and retrieval flows.

```bash
cd "$VELLUM_WORKSPACE_DIR"
# 1a. Screen for secret-bearing FILE NAMES; review every hit with the user.
find /path/to/raw-corpus \( -name '.env*' -o -name '*.key' -o -name '*.pem' \
  -o -name '*credential*' -o -name '*secret*' -o -name 'cookies*' \
  -o -path '*tokens*' -o -path '*oauth*' \) -print

# 1b. Screen file CONTENTS for credential shapes. --hidden and --no-ignore
#     matter: rg skips dotfiles and gitignored paths by default, which is
#     exactly where credentials live. Capture the FULL list (no truncation):
#     every file named here must be excluded below or cleaned with the user
#     before it lands.
rg -l -i --hidden --no-ignore \
  "api[_-]?key|access[_-]?token|refresh[_-]?token|client[_-]?secret|password\s*[=:]|passwd|bearer |AKIA[0-9A-Z]{16}|BEGIN [A-Z ]*PRIVATE KEY" \
  /path/to/raw-corpus > /tmp/corpus-secret-hits.txt
cat /tmp/corpus-secret-hits.txt

# 2. Build rsync exclusions from the content hits (paths relative to the
#    corpus root), then copy with ALL flagged paths excluded.
sed 's|^/path/to/raw-corpus/||' /tmp/corpus-secret-hits.txt > /tmp/corpus-secret-exclusions.txt
mkdir -p imports/<source>
rsync -a --exclude='.env*' --exclude='*.key' --exclude='*.pem' \
  --exclude='tokens/' --exclude='oauth/' --exclude='cookies*' \
  --exclude-from=/tmp/corpus-secret-exclusions.txt \
  /path/to/raw-corpus/ imports/<source>/
```

Rules:

- **Never place raw corpus files under `memory/`.** The cold store is `imports/<source>/`; the map is the only thing that enters memory.
- **Never land credentials in the cold store.** Exclude secret-bearing files during the copy; if the content screen finds embedded live tokens inside otherwise-wanted files, pause and resolve them with the user before landing those files.
- Treat the cold store as read-only once landed. The map pages and the drill-in skill both point at these paths; moving files later breaks every pointer.
- If the corpus is huge, check free disk space first and copy in batches. Prefer copy over move until the user confirms the original can be released.

### Step 3: Inventory and slice plan

Census the corpus and produce a machine-readable slice plan:

```bash
mkdir -p "$VELLUM_WORKSPACE_DIR/imports/<source>/.staging"
bun run {baseDir}/scripts/inventory.ts "$VELLUM_WORKSPACE_DIR/imports/<source>" \
  > "$VELLUM_WORKSPACE_DIR/imports/<source>/.staging/plan.json"
```

The script prints a human census to stderr (file count, total size, extension mix, date range) and a JSON plan to stdout: `{ files, totalBytes, byExtension, dateRange, suggestedSlices }`. Each suggested slice is a date-windowed group of files sized for one skim pass. Review the plan before skimming:

- If the slice count is outside roughly 10 to 50, adjust: merge sparse adjacent slices or split dense ones. The plan is a suggestion, not a contract.
- Files with no recognizable date cluster on file mtime; if mtimes are all import-day (a fresh copy), pick slices by directory or topic instead and say so in the map.

### Step 4: Skim each slice into a staged map page

For each slice in the plan, run one summarization pass that reads the slice's files and writes one staged map page:

- Output goes to the staging directory as `<slug>.md` (for example `imports/fathom/.staging/fathom-recordings-2025-q1.md`). The slug is the filename minus `.md`.
- Every page follows `references/map-page-template.md` exactly: lead that stands alone as the retrieval card, a `Raw data:` pointer line in the lead, `## ` sections per topic or time slice, and `source:` / `origin_date:` / `ref_files:` / `links:` frontmatter.
- Also write one corpus index page (`kind: index`) whose `links:` enumerate all slice pages.

**Bound the fan-out.** The number of skim passes equals the number of slices in the plan, full stop. Run them a few at a time (parallelism of 3 or 4 is plenty). Never spawn one pass per file, never let a pass recursively spawn more passes, and never re-skim slices that already have a staged page unless their content changed. An unbounded fan-out over a large corpus is the expensive failure mode of this skill.

A skim pas