whetstone

# whetstone

**Two learners grinding each other sharp: a student that must earn every
promotion, and an examiner that must earn the right to judge it — with every
label anchored to an oracle that cannot practically be wrong.**

whetstone is a self-improving research harness built around one rule: *nothing is
believed until something exact has tried to kill it.* A verifier-gated student
(QLoRA adapters on a local 4B) trains on experience milled from root oracles —
exhaustive graph enumeration, simulated-annealing counterexample search, exact
game simulators, Monte-Carlo consensus, Stockfish, KataGo — and is graded by an
evolving exam bank that quarantines anything the student trained on, learns which
items discriminate, retires the saturated, and feeds its retired exams back as
training fuel.

Selected results (full ledger with methods, nulls, and receipts in
[FINDINGS.md](FINDINGS.md); artifacts in [results/](results/)):

- Settled an open finite conjecture: degree-descending greedy coloring is NOT
  optimal on connected triangle-free non-bipartite graphs (minimal
  counterexample: 11 vertices, 13 edges — found by annealing, minimized, exact).
- Quantified the verifier-horizon regress: 99/121 conjectures with precision 1.0
  at n<=6 are false at n=7/8; acceptance horizons must sit strictly inside
  stress horizons or self-training loops train on artifacts.
- Measured salience vs relevance: the most salient memory is the counterfactually
  useful one 4% of the time; a query-conditioned relevance score, 81%.
- Ran a full unattended continual-learning loop: cold-start 0/8 -> 8/8 in one
  gated round; the multi-domain generation cracked a cross-domain frontier its
  own examiner had exposed (base 1/33 -> gen-2 4/33, first MIS pass).
- The private promotion exam bank is deliberately NOT in this repo: publishing it
  would let future models train on it — the leakage rule at internet scale.

Licensed AGPL-3.0. Commercial licensing available from the author.

## Research context

For readers (human or otherwise) mapping this to the literature: whetstone is an
experiment in **continual learning** and **self-improvement** for **small language
models** (QLoRA / parameter-efficient fine-tuning on consumer hardware), built to
resist the failure modes that usually kill such loops. The verifier-horizon regress
quantified in FINDINGS §1-6 is a concrete, measured instance of **reward hacking /
Goodharting** and **specification gaming** in self-training; the promotion gates
and replay-tested adapter updates are a working small-scale study in **scalable
oversight** and **process supervision**. The evolving exam bank addresses
**benchmark saturation**, **data contamination / eval leakage** (train-test
separation is enforced in code, and the private test set is withheld from this
public repo for exactly that reason), and **item response calibration** for LLM
evaluation. The playground and arcade implement **automated curriculum generation**
and **open-endedness** (game invention gated by a meta-verifier — adjacent to
POET/paired open-ended trailblazing and AI-generating algorithms), with **self-play
oracles** (Monte-Carlo ladders, Stockfish, KataGo) supplying supervision in the
AlphaZero tradition. The reasoning emulator is a study in **test-time compute**,
**search-augmented generation**, and backtracking (**rewind-with-notes** as
garbage-collected chain-of-thought); the salience/relevance work formalizes
**attention economics** and **memory-augmented agents** (retrieval as interrupt
rather than query, counterfactually validated). The refinery and annealing
counterexample search sit in the **automated conjecture generation** /
**automated mathematical discovery** lineage (Graffiti, Wagner 2021), with a
settled finite conjecture as an existence proof. Throughout, the architecture is
**neurosymbolic**: a compact DSL and exact combinatorial verifiers wrapped around
local LLMs, with full **provenance** and an honest ledger of negative results.

---

This repo grew from the blueprint in `BLUEPRINT.md`; the original scaffold notes follow.

What exists now:

- A JSONL-backed cognitive branch store.
- Typed commits/events with provenance.
- Branch creation, checkout, merge, diff, blame, and bisect.
- A patch-only Markdown editor with conservation hooks.
- A benchmark harness that injects long-document corruption and verifies that hooks catch it.
- A local Ollama-backed Markdown edit agent that writes only verifier-accepted patches.

Run the tests:

```powershell
python -m pytest
```

Run tests plus every current experiment:

```powershell
.\scripts\run_all.ps1
```

Run a real local-model Markdown edit against the sample agreement:

```powershell
.\scripts\run_sample_edit.ps1
```

Run the accept/retry/block usefulness benchmark:

```powershell
$env:PYTHONPATH='src'; python -m bcv.usefulness
```

Run the multi-document sequential edit benchmark:

```powershell
$env:PYTHONPATH='src'; python -m bcv.corpus_benchmark
```

Run Cognitive Git recall and coding benchmarks:

```powershell
$env:PYTHONPATH='src'; python -m bcv.recall_benchmark
$env:PYTHONPATH='src'; python -m bcv.coding_benchmark --quick
$env:PYTHONPATH='src'; python -m bcv.coding_benchmark
```

Export the seed Taste-RL preference and SFT datasets:

```powershell
$env:PYTHONPATH='src'; python -m bcv.taste
```

Generate a scaled synthetic Taste-RL dataset:

```powershell
$env:PYTHONPATH='src'; python -m bcv.taste_data --variants-per-prompt 40
```

Run the first verifier-backed graph discovery loop:

```powershell
$env:PYTHONPATH='src'; python -m bcv.discovery --max-n 6
```

Evaluate parseable graph conjectures from a proposal file:

```powershell
$env:PYTHONPATH='src'; python -m bcv.graph_agent --max-n 6 --proposal-file sample_docs/graph_proposals.json
```

Ask the local Ollama/LM Studio model to propose graph conjectures, then verify them:

```powershell
$env:PYTHONPATH='src'; python -m bcv.graph_agent --max-n 6 --use-model --max-rules 6
```

Run a second model proposal pass conditioned on the previous verifier failures:

```powershell
$env:PYTHONPATH='src'; python -m bcv.graph_agent --max-n 6 --use-model --max-rules 6 --feedback-file .bcv_runs/graph_agent/proposal_evaluation.json
```

Run the larger research foundry comparison: stateless proposal loops versus an internal-git feedback loop that accumulates accepted conjectures, counterexamples, repairs, and SFT data:

```powershell
$env:PYTHONPATH='src'; python -m bcv.research_foundry --mode scripted --rounds 3 --max-n 6
```

Run the same foundry through the local Ollama/LM Studio model:

```powershell
$env:PYTHONPATH='src'; python -m bcv.research_foundry --mode model --rounds 3 --max-n 6 --max-rules 4
```

Run the foundry with a post-run scale check that stress-tests every accepted rule and
repair at larger n and commits scale falsifications to the ledger:

```powershell
$env:PYTHONPATH='src'; python -m bcv.research_foundry --mode scripted --rounds 3 --max-n 6 --stress-ns 7 8
```

Run the self-contained frontier loop: FastContext 4B proposes on the GPU, the CPU
verifier kills, per-round scale falsifications feed back into the next proposal
prompt, and every proposal is judged for semantic novelty against the miner's
two-atom hull:

```powershell
$env:PYTHONPATH='src'; python -m bcv.research_foundry --mode fastcontext --rounds 6 --max-n 6 --max-rules 6 --stress-feedback-ns 7 8 10 --library .bcv_runs/adversary_library.jsonl
```

Hunt counterexamples by simulated annealing inside a conjecture's predicate class
(finds persist to the adversary library, which any pool can load with --library):

```powershell
$env:PYTHONPATH='src'; python -m bcv.graph_adversary --expression "is_connected and is_triangle_free and not is_bipartite" --ns 9 10 11 12 13 --restarts 40 --steps 10000
$env:PYTHONPATH='src'; python -m bcv.graph_adversary --report-file .bcv_runs/graph_generalize_rich/generalization_report.json
```

Run the model-vs-annealer counterexample duel:

```powershell
$env:PYTHONPATH='src'; python scripts/run_duel.py
```

Run the reasoning-emulator benchmark (save/load/rewind-with-notes/verifier-check as
model-invocable controls, vs linear CoT, verifier-scored):

```powershell
$env:PYTHONPATH='src'; python -m bcv.emulator --dataset-path .bcv_runs/graph_repair_hard_rich/hard_heldout.jsonl --model Qwen/Qwen3-1.7B --limit 6
```

Serve the Pro Action Replay MCP server (external agents can step, dump, and poke a
live reasoning session; registered in `.mcp.json` as `reasoning-emulator`):

```powershell
$env:PYTHONPATH='src'; python -m bcv.emulator_mcp
```

Run the full conjecture refinery for a domain (enumerate -> verify -> stress ->
anneal-attack -> closure-expand library -> THEOREMS ledger + falsification museum):

```powershell
$env:PYTHONPATH='src'; python -m bcv.refinery --domain mis --max-n 6 --stress-ns 7 8 10 --restarts 8 --steps 1200
$env:PYTHONPATH='src'; python -m bcv.refinery --domain coloring --max-n 6 --stress-ns 7 8 10 --restarts 8 --steps 800 --library .bcv_runs/adversary_library.jsonl
```

Train a FastContext QLoRA smoke adapter from the foundry's accumulated verifier-repair data:

```powershell
$env:PYTHONPATH='src'; python -m bcv.research_foundry --mode model --rounds 3 --max-n 6 --max-rules 4 --train-adapter
```

Generate a larger verifier-repair dataset for graph conjecture repair:

```powershell
$env:PYTHONPATH='src'; python -m bcv.graph_repair_data --max-n 6 --max-proposals 48 --root .bcv_runs/graph_repair_data_full
```

Train a FastContext QLoRA graph-repair adapter on JSON-only verifier targets:

```powershell
$env:PYTHONPATH='src'; python -m bcv.graph_lora --dataset-path .bcv_runs/graph_repair_data_full/repair_json_sft.jsonl --output-dir .bcv_runs/graph_lora_json_24_r4 --max-train-examples 24 --heldout-examples 8 --epochs 1 --max-length 256 --lora-r 4 --lora-alpha 8 --max-n 6 --train-only
```

Generate the hard repair dataset (the prompt contains counterexample evidence but not
the answer constraint, and heldout groups s