vision-skills

# vision-skills

Five local CLIs that give a text-only agent eyes. They read one shared
vision config (`VISION_API_KEY` / `VISION_BASE_URL` / `VISION_MODEL` /
`LANG`), plus the optional Python-client settings `VISION_API_PROTOCOL`,
`VISION_REASONING_EFFORT`, and `VISION_USER_AGENT` — no extra credentials.

Pick the tool by the question you are answering:

| Question | Tool |
|---|---|
| "What does this image show / say?" | `glance` |
| "Where is X?" — a thing you can name | `ground` |
| "Where are all the Xs?" — every instance of a kind | `detect` |
| "What is its exact shape, size, offset?" | `trace` |
| "Cut this box out as its own image file" | `crop` |
| "OCR this long screenshot / scrolling page / chat history" | `scripts/long_screenshot_ocr.py` |
| "Extract the icon/logo foreground as transparent PNG — manual region or auto (cropped+scaled screenshots)" | `scripts/extract_fg.py` |
| "Turn this HTML file into a viewport or full-page screenshot" | `scripts/html_shot.py` |
| "Which colours dominate a region, and which palette value fits it?" | `scripts/dominant_colors.py` |
| A relation none of them return — a gap, a distance between two located things | code over the pixels (Pillow) |

`glance` answers what something is; `ground` and `detect` answer where.
You give `ground` a description of a particular thing; you give `detect` a
kind and it enumerates the instances.

Both give real coordinates, but they are not pixel-exact: the box arrives
on a 0-1000 grid and is scaled to your image, so the last pixel or few are
not reliable. That is accurate enough to crop with, to click, to compare
positions against. When a number has to be exact, `trace` derives it from
the actual pixels — offsets, sizes, shapes.

## Use the provided tools before hand-rolled pixels

Everything this toolkit ships a tool for, call the tool — do not rewrite
it with Pillow in the middle of a task. The CLIs exist so the same pixel
work is not hand-coded differently every time:

- cut a box out of an image → `crop`, not `Image.open(...).crop(...)`
- sample a region's palette → `scripts/dominant_colors.py`
- compare two images → `scripts/pixel_diff.py`
- vectorize to SVG → `trace`
- locate / inventory elements → `ground` / `detect`
- describe / OCR an image → `glance`
- safely split, OCR, and merge a long screenshot → `scripts/long_screenshot_ocr.py`
- HTML file to a viewport or full-page screenshot → `scripts/html_shot.py`

Hand-written Pillow is only for what none of them return: a relation
between two things you already located (a gap, a distance), a resize or
overlay, drawing. If you catch yourself writing `.crop()`, `.convert()`,
or histogram code where one of the tools above fits, replace it with the
tool call — same coordinates, same box format, and the output feeds the
next tool directly.

## glance — ask about an image

```bash
glance <image>                                 # detailed description
glance <image> -q "<question>"                 # targeted question (qualitative only)
glance <image> --ocr                           # verbatim OCR
glance <image> --region X1,Y1,X2,Y2 -q "..."   # zoom into a crop
glance <img1> <img2> -q "..."                  # compare in ONE call
```

When you do compare with `glance`, pass all paths to one call — separate
calls cannot see both images, so two descriptions compared afterwards are
two hallucination surfaces, not a comparison. `--region` uploads only the
crop, so small text and icons become readable.

But "what changed between these two?" is not a glance question. A one-word
badge or a small shift is a rounding error to a vision model and exact to
`scripts/pixel_diff.py`. Diff first to get the box, then `glance --region`
that box to read what the change actually is.

For a tall scrolling screenshot, do not send the whole image through one OCR
call and accept the model's downscaling loss. Run the long-screenshot workflow,
which finds low-content cut bands, invokes `glance` on each chunk, uses
structured extraction for chat histories, merges only duplicated overlap, and
writes a boundary audit:

```bash
python3 scripts/long_screenshot_ocr.py work/page.png -o work/page.ocr.md
python3 scripts/long_screenshot_ocr.py work/chat.png --mode chat --resume -o work/chat.ocr.md
```

Read `references/long-screenshot-ocr.md` before using it. It defines the
verification pass for unsafe cuts and chat-message boundaries.

## ground — locate a named target

```bash
ground <image> "<target description>"
ground <image> "<target>" --region X1,Y1,X2,Y2
```

Output: `x1: .., y1: .., x2: .., y2: ..` in original-image pixels — with
`--region` too (crop hits are mapped back).

Provider-native 0-1000 boxes do not all use the same array order: Gemini uses
`[y0, x0, y1, x1]`, while Qwen3-VL, Qwen3.5, and Qwen3.6 use
`[x0, y0, x1, y1]`. Grounding code must select the order by model family (or
an explicit override) before scaling to pixels; never parse every provider as
Gemini-style `yxyx`.

If several boxes come back numbered, your description matched more than
one element rather than picking out a single thing. Narrow it with what
distinguishes the one you mean — its text, its position, the block it sits
in — and ask again.

The box is a handle, not just an answer — it feeds the next call:

```bash
$ ground screenshot.png "the send button"
x1: 1067, y1: 841, x2: 1108, y2: 881
$ glance screenshot.png --region 1067,841,1108,881 -q "is it enabled or greyed out?"
```

That two-step is how you inspect anything too small to survive a
full-image pass.

## detect — find every instance of a kind

```bash
detect <image>                        # every UI element
detect <image> "buttons"              # one kind only
detect <image> --region X1,Y1,X2,Y2   # inside one box
```

You name a particular thing for `ground`; you name a kind for `detect` and
it enumerates the instances. Output is a numbered list with each item's
visible text and box. A full-screen
pass is a fast first draft — counts vary run to run on