release-sample-sweep

# Release sample sweep (chrome-devtools)

Before a release, every user-facing surface is exercised the way a user
exercises it. This skill is that procedure written down.

Three surfaces, three drivers — all three are release gates:

| Surface | What | Driver |
|---|---|---|
| **Samples** (33) | `samples/*`, booted from packaged artifacts | chrome-devtools MCP, or the wire protocol for the headless ones |
| **Expo client** (1) | `samples/spring-boot-ai-classroom/expo-client/` | iOS simulator MCP — it is a native app, chrome-devtools cannot reach it |
| **CLI** | `atmosphere run` / `new` / `compose` / `import` / `checkpoint` + its four distributions | Shell, then chrome-devtools against what `atmosphere run` booted |

## When to run it

- **Before cutting any release.** Non-negotiable — it is the last gate before
  `release-4x.yml`.
- After a change to the **Console bundle** (`modules/spring-boot-starter/frontend/`),
  since the Console is both the shipped sample UI and the validation surface.
- After a change to a **shared module** that every sample transitively depends on
  (`modules/cpr`, `modules/ai`, `modules/spring-boot-starter`, `modules/admin`).
- After a **dependency bump wave** — two of the last three sweeps found a
  version-skew bug that compiled clean and only failed at runtime.

## What it catches that CI does not

CI builds and tests modules; this sweep exercises **packaged artifacts in a
browser**. The gap between those is where the real bugs live:

| Sweep | Bug found | Why CI was green |
|---|---|---|
| 2026-06-30 | `quarkus-ai-chat` would not start — OTel api/common version skew from a Dependabot bump | Module tests never boot the sample's fast-jar |
| 2026-07-17 | `spring-boot-orchestration-demo` crashed on every tool turn — the sample pom hardcoded `langchain4j-open-ai:1.15.0` while the reactor manages 1.17.0 | The module built against 1.17.0; only the sample's own jar bundled 1.15.0 |

`scripts/release-gate-samples.sh` automates the boot-and-assert half of this in
CI. This sweep is the **browser half** on top of it — the layer that sees
rendering, streaming, transport headers, tool cards, and console errors.

## The shape of the sweep

```
Step 0   Preconditions  — build everything, start Ollama, free the ports, open the ledger
Step 1a  Samples        — 33 samples: launch → drive → collect → verdict → teardown
Step 1b  Expo client    — the RN client in the iOS simulator
Step 1c  CLI            — atmosphere run/new/compose/import/checkpoint + distributions
                           ALL OF PHASE 1 IS COLLECT-ONLY. Do not fix anything mid-sweep.
Step 2   Triage         — classify every finding, rank by blast radius
Step 3   Fix            — root-cause fix + a regression test per issue, in the right
                           suite, each proven to bite
Step 4   Re-test        — the failed surfaces in full, plus the blast-radius subset
                           of already-passing ones
Step 5   Report         — vault report, CI green, memory updated
```

Step 1 is deliberately fix-free. Fixing mid-sweep changes the artifact under
test and invalidates every sample already verified against the old one. The one
exception: a defect that **blocks the sweep itself** from continuing — fix it,
say so in the ledger, and note which already-passed samples were re-run.

## Non-negotiables

1. **chrome-devtools, never curl, for validation.** `curl` is allowed only for
   port readiness and for headless wire protocols (A2A/MCP/REST) that serve no
   HTML. A "works via curl" claim skips the whole JS layer and is a false pass.
2. **The Atmosphere Console is the UI.** Drive `/atmosphere/console/` (Spring
   Boot samples redirect `/` there). A sample that needs a bespoke page instead
   of the Console is itself a finding.
3. **Assert the rendered element, not the payload.** An `image` node with a
   `src` is a rendered screenshot; the same base64 in a `StaticText` node means
   nothing rendered it. "Server started", "HTTP 200", and "bytes present in the
   DOM" are not passes.
4. **Boot the packaged artifact.** `java -jar` (or `quarkus-run.jar`), never
   `spring-boot:run` / `quarkus:dev`. Both historical bugs above existed *only*
   at artifact level.
5. **Kill by PID, never `pkill -f`.** Never touch a port or process the sweep
   did not start — if a port is occupied, move to another port. The same rule
   covers the machine's **network**: never run `networksetup`, never take an
   interface down, never touch VPN/DNS/proxy settings. The host's Wi-Fi carries
   every session the maintainer has open, and a sweep interrupted mid-toggle can
   leave the machine offline indefinitely. Any assertion that needs real network
   loss is recorded PARTIAL with its unit coverage cited — see
   `references/expo-sweep.md`.
6. **Model limitation ≠ framework bug.** A small local model emitting invalid
   tool-call arguments is a model limitation; record it as such and prove it by
   re-running the same flow on a capable model before calling it a regression.
7. **Never write "flaky".** Reproduce it, or explain the mechanism. If neither
   is possible yet, it is a FAIL with an open question, not a dismissal.
8. **Report honestly.** PASS / PARTIAL / FAIL with one line of concrete
   evidence each. PARTIAL must name what was not proven and why.

## Step 0 — Preconditions

```bash
git status --porcelain                    # must be clean
git rev-parse --short HEAD                # record this SHA in the ledger
grep -m1 '<version>' pom.xml              # record the version under test

./mvnw install -DskipTests -Pfastinstall  # full reactor: framework + every sample jar
./scripts/sync-console-bundle.sh --check  # the Console you will drive must be current

ollama list                               # qwen2.5:3b + qwen2.5:7b-instruct-q4_K_M
curl -s -o /dev/null -w '%{http_code}\n' http://localhost:11434/v1/models
```

- **LLM backend is local Ollama, keyless.** Use `qwen2.5:3b` for streaming
  samples and `qwe