simframe

# simframe

[![ci](https://github.com/lvlrSajjad/simframe/actions/workflows/ci.yml/badge.svg)](https://github.com/lvlrSajjad/simframe/actions/workflows/ci.yml)
[![npm](https://img.shields.io/npm/v/simframe.svg)](https://www.npmjs.com/package/simframe)
[![license](https://img.shields.io/npm/l/simframe.svg)](./LICENSE)

**Eyes, hands and memory for an agent driving the iOS Simulator.**

[Website](https://lvlrsajjad.github.io/simframe/) · [npm](https://www.npmjs.com/package/simframe)

An agent driving the iOS Simulator is slow for three reasons, and only the first
one is obvious:

1. **Every look is a wait.** `simctl io screenshot` costs ~130 ms of blocking
   latency, paid again on every glance — and paid twice whenever the agent
   captures mid-animation and has to look again.
2. **Every step is a round trip.** Tap, screenshot, reason, tap, screenshot. A
   twelve-step flow costs twelve model turns, and the model turns cost far more
   than the milliseconds.
3. **Nothing is remembered.** The same screen gets re-read and re-reasoned about
   every single time it appears.

simframe attacks all three: a background loop keeps the newest frame warm, whole
flows run in one call, and screens the agent has seen before are answered from
memory.

## What changed, measured

Same four-tab navigation flow, on a real production app:

| | Before | With simframe |
| --- | --- | --- |
| Look at the screen | ~130–400 ms, blocking | **~20 ms**, already captured |
| "Did anything change?" | a full image | **~2 ms**, text only |
| A 5-step flow | 5+ model round trips | **1 call**, ~7 s |
| Finding a control | read tree (~570 ms) + reason | **~1 ms** from memory |
| Same flow, 3rd run | no improvement — every run is the first | **3304 ms, 4/4 from memory** |

The same four-tab tour, run three times back to back: **7370 ms → 5160 ms →
3304 ms**, with 1, then 3, then 4 of the four controls resolved from memory and
no mis-taps. What is left is mostly the app's own animation and data load.

Those figures are with per-step verification **off**. It is now on by default,
and it is not free: the same tour runs ~30 s, ~29 s, ~27 s, resolving 4/4
controls from memory on every pass and verifying 4/4 steps from the second pass
on. The trade is a flow that tells you when a step did not do what you meant
against a flow that is faster and does not. Pass `verify: false` for the old
behaviour; the reasoning, and the cost breakdown, are in
[`docs/BENCHMARKS.md`](docs/BENCHMARKS.md).

## Install

```bash
npm install -g simframe
simframe doctor
```

The first `simframe start` builds a small Swift daemon from source — a few
seconds, once. It needs the Xcode command line tools, which you already have if
you have a simulator. Without them simframe falls back to the original
`simctl` loop and says so.

`doctor` checks each capability separately and tells you what you have:

```
ok   xcrun              xcrun version 72.
ok   sips               available
ok   input driver (idb) companion built Sep 1 2026
ok   on-device OCR      available
ok   booted simulator   iPhone 17 Pro (iOS 26.5)
ok   capture            frame #888 322x700 in 2ms (age 538ms)
```

### Claude Code

```bash
claude mcp add --scope user simframe -- npx -y simframe mcp
```

`--scope user` makes it available in every session; without it the server is
registered only for the directory you ran the command in.

### Any other MCP client

```json
{
  "mcpServers": {
    "simframe": { "command": "npx", "args": ["-y", "simframe", "mcp"] }
  }
}
```

## Capabilities are independent

Each layer works without the ones above it, and `doctor` tells you which you
have. **Observation needs nothing but Xcode.**

| Capability | Needs | Without it |
| --- | --- | --- |
| Watch the screen, wait, recall | nothing extra | — |
| Read labels + coordinates from pixels | `swiftc` (Xcode CLT) | falls back to the accessibility tree alone |
| Tap, type, swipe | nothing extra, or [`idb`](https://fbidb.io) as a fallback | simframe observes but cannot touch |
| Accessibility tree | [`idb`](https://fbidb.io) | OCR alone still yields labels and coordinates |

`simframe doctor` names which engine is carrying each capability, per device.
**idb is now required only for the accessibility tree** — capture, input and text
recognition all run in-process.

```bash
# input, optional
brew tap facebook/fb && brew install idb-companion && pipx install fb-idb
```

Homebrew may ask you to trust the tap first; that is a deliberate prompt for a
human, and the narrow form is `brew trust --formula facebook/fb/idb-companion`.

## The tools

| Tool | What it does |
| --- | --- |
| `sim_look` | Newest frame as an image, no capture wait. |
| `sim_state` | Text only: screen hash, what changed **since your last look**, region movement map. |
| `sim_wait` | Waits for the screen to change *and then* settle. |
| `sim_do` | A whole flow in one call — tap, type, scroll, assert — each step settling before the next. |
| `sim_ui` | The screen as labels + tap coordinates, from accessibility **and** OCR. |
| `sim_recall` | Look backwards: a timeline of what happened, or the frame from N seconds ago. |
| `sim_strip` | Recent frames tiled into one image. |
| `sim_capture` / `sim_devices` | Manage capture loops; list simulators. |

## Baselines: the thing to understand

Every change question is really "changed **since when**?" — and the answer is
almost never "since the previous frame". A UI transition is over in about 700 ms,
so comparing consecutive frames tells a caller that polls every few seconds
"nothing changed", even though the screen is completely different from when it
last looked.

So simframe compares against **the last frame you observed**. Over MCP that is
automatic. From the CLI, capture a baseline before you act:

```bash
H=$(simframe mark)
# ...tap, launch, navigate...
simframe wait --since=$H     # change, then settle
simframe state --since=$H    # what moved, as text
```

The same applies to waiting. `--mode=settle` (the default) waits for a change and
*then* for stillness, because a bare "wait until stable" called in the moment
before an animation starts will correctly, and uselessly, return immediately.

## Screen memory

An accessibility tree is a promise apps do not always keep. In testing against a
real production app, its custom tab bar published **no children at all**, its
icon buttons carried unreadable private-use glyphs, and its React Native text
inputs were **absent from the tree entirely** — the controls used most were
exactly the ones that could not be tapped by name.

So simframe reads the screen two ways and remembers the result:

- **Accessibility** gives real hit targets, types and enabled state.
- **On-device OCR** (Apple's Vision, ~290 ms, no model round trip) gives every
  label a person can actually see, with coordinates.
- The merge is keyed by a **structural fingerprint**, so the next visit is a
  file read. What that fingerprint is, and why it is not a pixel hash, is below.

```
first visit to a screen   ~1000 ms   read tree + OCR, store the map
every visit after that       ~1 ms   look it up
```

OCR is also more accurate than measuring by eye. On one tab bar the first tab
centre sat at x=62, not the x=40 an even five-way split predicts — a silent
mis-tap on every attempt.

Two details that matter:

- **Containers do not absorb their contents.** A tab bar encloses all five tab
  labels but is not any of them, so the merge only combines an element with text
  of comparable size.
- **Ambiguity is reported, not guessed.** A word that is both a screen title and
  a tab returns an error listing both with coordinates, because silently tapping
  the title looks exactly like nothing happening.

### Two hashes, because there are two questions

"Did this move?" and "is this the same screen?" look like one question and are
not. simframe answers them separately, and getting that wrong was the single
most expensive mistake in its development.

**Change and settle** are questions about pixels, so a pixel hash answers them.
The frame hash changes whenever any pixel group changes — a clock digit, one new
row — which is exactly right for "did anything happen?" and useless as a key for
"have I been here before?". For change detection there is a layout hash: status
bar cropped, difference hash over a 12×24 grid.

A mean-threshold hash was tried first and was actively dangerous: low-contrast
screens collapsed onto identical values, so unrelated screens matched at distance
0 and taps landed on the wrong control. The difference hash fixed that.

**Identity is not a question about pixels**, and this is the part that took three
attempts. Content *is* pixels: a list whose rows changed drifts as far as a
different screen does. Measured on a real app, same-screen revisits reached 62
bits against a different-screen floor of 74 — overlapping, with no threshold
available to choose. An earlier calibration had suggested a comfortable margin
(0–4 against 77–113), but it was measured on screens whose content happened to be
stable and did not survive contact with a real list.

So identity is **structural**. The fingerprint is built from element roles,
frames quantised to a 24 px grid, the region each element sits in, and repeated
siblings bucketed as "one" or "many" rather than counted. Deliberately included:
the labels of chrome elements only — nav title, tab labels, toolbar buttons —
because two list screens with identical structure are told apart by their title
and nothing else. Deliberately excluded: all content text and values, the status
bar, and the keyboard region when a keyboard is up.

It does not depend on the accessibility tree. Fingerprinting from OCR boxes
alone, with the tree discarded entirely, still separates screens — different
screens ceiling 0.35 against the same threshold.

| | Jaccard similarity |
| --- | --- |
| Same screen, revisited | 0.41–1.00 |
| **Different screens** | **0.00–0.31** |

The threshold sits in that gap. It is