octoperf-validation-triage

# OctoPerf — Validation-failure triage

When a Virtual User validation finishes with many failing actions,
reading each failure detail one by one wastes context window and time.
This skill groups failures by **root cause**, drills into one
representative per group, and maps each group to the correct fix.

## When this applies

- A `validate_virtual_user` run finished with `finished=true` and at least one failing action.
- The user wants to know *what's wrong*, not just retry blindly.

## Steps

### 1. Get the failure index — no bodies yet

```
mcp__octoperf__get_virtual_user_validation_index(virtualUserId)
```

The index returns one entry per **validated** HTTP action — `actionId`,
`success`/`total` counts, `successTimestamps` and `failedTimestamps`. For
triage, focus on the entries whose `failedTimestamps` is non-empty (the
actions that failed at least one run); entries with only `successTimestamps`
passed every run. **Do not fetch failure details yet** — the index alone is
usually enough to classify.

(A `successTimestamps` entry is also a handle to read a *passing* run's
body: pass it to `fetch_validation_http_body` with
`kind=VALIDATION_RESPONSE` — e.g. to inspect what a Debug-style action
actually captured on a green run.)

OctoPerf's "KO" rule isn't just "non-2XX". A sample is KO when:

| Live response code | Recorded response code | Result                                                           |
|--------------------|------------------------|------------------------------------------------------------------|
| 2XX                | none                   | ✅ OK                                                             |
| 2XX                | 2XX                    | ✅ OK                                                             |
| 2XX                | 3XX                    | ❌ KO (recording expected a redirect, got a body — usually wrong) |
| 3XX                | 3XX                    | ✅ OK                                                             |
| 3XX                | 2XX                    | ❌ KO (recording expected a body, got a redirect)                 |
| Any 4XX / 5XX      | anything               | ❌ KO                                                             |
| Unknown code       | anything               | ❌ KO                                                             |
| Any code           | 4XX / 5XX / unknown    | ❌ KO (recording was already broken; re-record)                   |

This matters when classifying: a "successful" 200 against a recorded 302 is a
real bug, not a false positive — the VU is hitting a different code path than
when it was recorded.

**Caveat — ResponseAssertion overrides the matrix.** A `ResponseAssertion`
attached to the action can mark a sample KO even when the matrix above
says OK (e.g. status 200 against recorded 200, but the body contains
the assertion's pattern — typical for forms that re-render with an
error message on a 200). Check for assertions on the failing action
before assuming an HTTP-level mismatch. The assertion's `type` field
also controls scope:

- `REQUEST_ONLY` — only the parent sample is checked.
- `REQUEST_AND_SUBREQUESTS` — parent **and** embedded resources. Matters when `downloadResources=true`: a 404 on an embedded CSS can fail the parent.
- `SUBREQUESTS_ONLY` — only embedded resources.

### 1b. Partial failures (`success < total`)

When the index entry shows `success < total` (e.g.
`{"success": 2, "total": 3, "failedTimestamps": [...]}`), the action
failed on **some iterations** but passed on others. Common patterns:

- **Bad row in a CSVVariable** — one iteration picks a row that
  doesn't pass server-side validation (e.g. a wrong-password row,
  a malformed account id, a stale token). Cheapest possible fix:
  edit the CSV. Investigate this **before** correlation when the VU
  uses a `CSVVariable`.
- **Race condition** — uncommon in 1-user validation but possible if
  the VU has internal `LoopContainerAction` or shared variables.
- **CSV exhausted with EOF = StopVU** — iterations past the CSV length
  never run; not strictly a failure but `success < total` reflects it.

`failedTimestamps` lists every failed iteration's epoch-ms. Pass the
**exact** timestamp to `get_validation_failure_detail` to pin the
failing iteration instead of getting a random one — critical when
debugging CSV-driven flakiness so you see the bad row, not a good one.

### 2. Group by category

Bucket the failing actions into a small number of categories. The
common ones:

| Category                  | Index signal                                                                                                                                      | Likely fix                                                                                                                                                                              |
|---------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| **Auth / state**          | 401, 403; "invalid token", "expired", "CSRF"                                                                                                      | Auto-correlation (separate skill)                                                                                                                                                       |
| **Variable / data**       | 400 with body validation errors; "field required", "invalid format"                                                                               | Edit / create variables; check CSV upload                                                                                                                                               |
| **HTTP server config**