sift

# sift

<!-- mcp-name: io.github.slymnysr/sift -->

Runs your command, then gives the model only the lines that matter.

A test suite prints 4,000 lines and eleven of them are the failure. A build
prints a progress bar that redraws 900 times. An install lists every package it
touched. All of it lands in the context window, and — this is the part that
costs — it is re-sent in full on every turn that follows.

`sift` is an MCP server and a command line for that problem. It runs the command
itself, keeps every byte on disk, and hands the model a view: the failures, the
summary, the lines a reader would actually stop on. The rest is marked, not
deleted.

Two numbers, both measured, both reproducible from this repository. Running this
project's own test suite prints 652 lines — 21,392 tokens, as the model's own
tokenizer counts them. What comes back is 8 lines and 208 tokens: **99% fewer**
(`python test/kazanc.py`). Removing almost everything is the easy half. Over a
22-sample corpus of real build and test output, the default budget keeps **138
of the 140 lines a reader could not do without** (`python test/budget.py`).

```
$ sift run -- pytest
FAILED test/test_auth.py::test_expired_token - assert 401 == 200
  ...
─ 3,914 lines not shown · sift peek 9f2c41ab for any of them ─
=== 1 failed, 212 passed in 18.4s ===
```

A file you did not produce — a log, a saved CI transcript, a crash dump — gets
the same treatment without being run:

```
$ sift digest ci-run-8812.log
2026-09-02T04:11:07 Building 214 targets
─ 38,904 lines not shown · sift peek ci-run-8812.log for any of them ─
ERROR: //src/parser:parse_test failed in 4.1s
─ 1,022 lines not shown · sift peek ci-run-8812.log for any of them ─
FAILED: 1 of 214 targets
```

A list is not read in lines. A JSON array written for a machine often has no
newlines at all, and there is nothing in a line of one worth choosing, so the
record becomes the unit and the same rules hold:

```
$ sift digest export.json
{"id": 3, "status": "failed", "error": "connection refused"}
─ 412 records not shown · sift peek export.json for the text they came from ─
{"id": 416, "status": "ok"}
```

The same question, asked about source code instead, is a table of
contents:

```
$ sift outline src/parser.rs
pub struct Parser {
pub fn parse(input: &str) -> Result<Ast, Error> {
─ 34 lines not shown · sift peek src/parser.rs for any of them ─
impl Iterator for Tokens {
```

And a command that does not end — a dev server, a log tail, a build you want to
keep working during — is started and then read a slice at a time:

```
$ sift run --background -- cargo build --release
9f2c41ab
$ sift follow 9f2c41ab
   warning: unused import: `std::fmt`
─ 212 lines not shown · sift peek 9f2c41ab for any of them ─
   error[E0308]: mismatched types
sift 9f2c41ab · running · new lines 1-247 · 2 shown · nemotron
$ sift follow 9f2c41ab          # only what has arrived since
$ sift follow --all --wait 30   # every run at once, holding for something new
$ sift stop 9f2c41ab            # ends it, and everything it started
```

Nothing is shown twice, and the numbers are the run's own: line 247 stays line
247 in `sift peek` for as long as the capture exists.

## What decides

A free model does. `sift` sends it the numbered lines and asks one question:
*which numbers matter?* It answers with numbers, and nothing else it says is
used — the text you read above is printed from the local capture, byte for byte.

That is the whole trick, and it is why this is not a summariser. A summariser
can be wrong about what a line said. `sift` cannot be: it never writes a line,
it only chooses one.

It also means language coverage is not a list. A model reads Turkish, Japanese,
Arabic and Hindi; it reads Rust, COBOL, Mojo and a language released last week.
Nothing here enumerates them, so nothing here can be missing one.

## Three rules

**Nothing shown is invented.** The judge returns line numbers. Text always comes
from the local file.

**Nothing is thrown away.** `sift peek <handle>` returns the raw capture,
unchanged. Every gap in a view says how many lines it covers.

**Nothing can break your command.** No API key, no network, an overloaded
endpoint, a nonsense reply, a bug in the distiller, or a decision that nothing
may leave this machine — every one of these falls back to rules that need none
of them. The command still runs; you still get its output; you get its exit
code.

## What leaves the machine

One thing: the text of a question. Before it is sent, anything credential-shaped
is replaced — tokens with a known prefix, JWTs, authorization headers, passwords
in connection strings, the body of a PEM block.

That costs you nothing to read. The model is only ever asked for line numbers,
and the lines are printed from your own file, so **a line masked on the way out
is still shown to you in full**.

Three switches:

```bash
SIFT_BASE_URL=... # ask your own endpoint instead, and no key is wanted
SIFT_MODELS=a,b   # which models to ask there, best first
SIFT_NO_MODEL=1   # never send anything; use the deterministic view
SIFT_MASK=0       # send unmasked
SIFT_CACHE=0      # ask again, even about text already answered
SIFT_EFFORT=low   # let the model think less, and lose some of what matters
SIFT_PATIENCE=0   # one quick pass only; do not wait out a busy hour
```

Masking is not complete and does not claim to be: a bare secret shaped like
nothing in particular gets through. `SIFT_NO_MODEL` is the one that guarantees.

The fourth is a measured trade and is off by default. Asked which lines matter
in a 404-line build, the model writes about 900 tokens of reasoning to produce a
twelve-token answer, and you wait 15 seconds for it. At `SIFT_EFFORT=low` the
same question takes 2.4 seconds — and over the corpus it loses 7.8% of the lines
a reader could not do without, in exactly the places this tool exists for: a
mainframe job's return code, a crash loop's diagnosis. Speed is available; it is
not the default, and the price is written down.

The third is about not paying twice. An answer already given for exactly these
bytes and exactly this question is used again instead of bought again — and what
is kept is the numbers, never the view, so the text is still rendered from your
own file and the gap still names your own capture. A view that cost no request
says `(remembered)` where it would otherwise name the model.

Captured bytes never leave `$SIFT_HOME` (`~/.cache/sift` by default). Nothing is
uploaded, nothing is logged elsewhere, and removing a capture directory removes
everything that was ever kept about it.

They also never go away on their own. Nothing here sweeps, expires or tidies in
the background: `sift gc [DAYS]` is the only thing that deletes a capture, and
it deletes when you type it and not before. What it leaves is one line per
handle — when it went and how big it was, never the command — so that a gap
marker read a fortnight later gets *"removed on the 8th"* instead of the answer
it would give for a handle you made up.

## Commands

```
sift run [--timeout SECONDS] [--shell] [--background] [--cwd DIR]
         [--budget LINES] [--keep PATTERN] [--] COMMAND...
sift follow [HANDLE] [--all] [--wait N]
                          what a background run has said since you last looked
sift stop [HANDLE]        end it, and everything it started
sift outline PATH         what a file declares, without its bodies
sift digest PATH...       what is in files somebody else produced
sift peek HANDLE|PATH [FIRST] [LAST]
sift hook                 answer one shell-command event on stdin
sift mcp                  speak the protocol on stdin, for a client
sift tools                which dense tools this machine has
sift tool NAME [ARGS...]  run one of them, distilled
sift memory [TERM]        what has been run here before, and how it went
sift list [COUNT]         what is running, and what has been run
sift stats [COUNT]        what the shortening cost, and what it saved
sift gc [DAYS]            remove captures older than that, and say what went
```

Several paths given to `digest` are asked about at the same time, and `--all`
follows every running command in one go. Both are the same idea: the waiting is
the cost, so do it once. `--wait N` holds until a run actually says something
instead of answering that nothing has happened yet.

`sift hook` is the way to catch the shell commands a client runs on its own,
without a proxy: point the client's pre-tool hook at it and every shell command
comes here first. **Everything is routed** — nothing here guesses which commands
are worth catching, because a list of those is a list of tools in disguise and
how much a command prints is not knowable before it runs. Routing everything
costs nothing: a view of twelve lines is twelve lines. It fails open, so a bug
in it leaves your shell exactly as it was, and `SIFT_HOOK=0` switches it off.

It is the part of this that pays most and the only part you have to turn on, so
it offers itself rather than waiting to be found:

```bash
sift hook --install      # says what it gives and what it costs, then asks
sift hook --uninstall    # and takes it back out
```

It writes one line into your client's settings, keeps a copy of the file as it
was, and touches nothing else that is in there. `sift run` mentions it once,
ever, and then stops.

`sift tool` runs one of three programs that answer a question *without opening
the file* — `sg` (ast-grep) for structural search, `diff` (difftastic) for a diff
that can tell a reindent from a change, `loc` (scc) for the size of a tree. Their
output is large by nature, which is exactly why they belong here. **No binaries
ship with this package**: `sift tools` says which of them this machine has and
what each is called, and installing one stays your decision.

`sift memory` asks no model at all. The question is counting — how often, how it
went, which command has never once worked here — and a model asked to count is
slower, costs a request