structured-debugging

# Structured Debugging

When debugging hard issues, the natural instinct is to form a theory and
immediately apply a fix. This fails more often than it works. The fix addresses
the wrong cause, adds complexity, creates false confidence, and obscures the
real issue. Worse, after several failed attempts you lose track of what's been
tried and start guessing randomly.

This methodology replaces guessing with a disciplined cycle that converges on
the root cause. Each iteration narrows the search space. It's slower per attempt
but dramatically faster overall because you stop wasting runs on wrong theories.

## The Cycle

### 1. Hypothesize

Before touching code, write down what you think is happening and why. Be
specific about the expected state at each step in the execution path.

Bad: "Something is wrong with the wait loop." Good: "The leader hangs because
`hasActiveTeammates()` returns true after all agents have reported completed,
likely because terminal status isn't being set on the agent object after the
backend process exits."

For bugs you expect to take more than one round, create a side note file for the
investigation in whichever location the project uses for such notes.

Write your hypothesis there. This file persists across conversation turns and
even across sessions — it's your investigation journal.

### 2. Design Instrumentation

Add targeted debug logs or assertions at the exact decision points that would
confirm or reject your hypothesis. Think about what data you need to see.

Don't scatter `console.log` everywhere. Identify the 2-3 places where your
hypothesis makes a testable prediction, and instrument those.

Prefer logging _values_ (return codes, payload contents, stream types, message
bodies, env state) over _presence checks_ ("was this function called?", "was
this branch taken?"). Code-path traces tell you what ran; data traces tell you
what it ran on. Most non-trivial bugs are correct code processing wrong data.

Ask yourself: "If my hypothesis is correct, what will I see at point X? If it's
wrong, what will I see instead?"

### 3. Verify Data Collection

Before running, confirm that your instrumentation output will actually be
captured and accessible.

Common traps:

- stderr discarded by `2>/dev/null` in the test command
- Process killed before flush (logs lost)
- Logging to a file in a directory that doesn't exist
- Output piped through something that truncates it
- Looking at log files from a _previous_ run, not the current one

A test run that produces no data is wasted.

### 4. Run and Observe

Execute the test. Read the actual output — every line of it. Don't assume what
it says.

When the data contradicts your hypothesis, believe the data. Don't rationalize
it away. The whole point of this step is to let reality override your theory.

### 5. Document Findings

Update the side note with:

- What the data showed (quote specific log lines)
- What was confirmed vs. disproved
- Updated hypothesis for the next iteration

This is critical for not losing context across attempts. Hard bugs typically
take 3-5 rounds. Without notes, you'll forget what you ruled out and waste runs
re-checking things.

### 6. Iterate

Update the hypothesis based on the new evidence. Go back to step 2. Each round
should narrow the search space.

If you're not making progress after 3 rounds, step back and question your
assumptions. The bug might be in a layer you haven't considered.

## Failure Modes to Avoid

These are the specific traps this methodology is designed to prevent. When you
notice yourself drifting toward any of them, stop and return to the cycle.

### Jumping to fixes without evidence

The most common failure. You have a plausible theory, so you "fix" it and run
again. If the theory was wrong, you've added complexity, wasted a test run, and
possibly introduced a new bug. The side note should always show "hypothesis
verified by [specific data]" before any fix is applied.

### Blaming external systems

"The model is hallucinating." "The API is flaky." "The library has a bug." These
conclusions feel satisfying because they put the problem outside your control.
They're also usually wrong.

Before blaming an external system, inspect what it actually received. A model
that appears to hallucinate may be responding rationally to stale data you
didn't know was there. An API that appears flaky may be receiving malformed
requests. Look at the inputs, not just the outputs.

### Inspecting code paths but not data

You instrument the code and prove it executes correctly — the right functions
are called, in the right order, with no errors. But the bug persists. Why?

Because the code can work perfectly while processing garbage input. A function
that correctly reads an inbox, correctly delivers messages, and correctly
formats output is still broken if the inbox contains stale messages from a
previous run.

Always inspect the _content_ flowing through the code, not just whether the code
runs. Check payloads, message contents, file data, and database state.

### Reframing the user's report instead of investigating it

When the user reports a symptom your own run doesn't reproduce, the
contradiction _is_ the evidence — the two environments differ in some way you
haven't identified yet. The wrong move is to reframe their report ("they must be
on a stale SHA", "they must be confused about what they saw", "must be a flake")
so that your run becomes the ground truth. Once you do that, every later piece
of evidence gets bent to defend the reframing, and the actual bug stays hidden.

The right move: catalogue what differs between their environment and yours (TTY
vs pipe, terminal emulator, shell, locale, env vars, prior state, build
artifacts) before forming any hypothesis. For ambiguous symptoms ("no output",
"it's slow", "it's wrong") ask one disambiguating question first — e.g., "does
does it hang or exit cleanly?" That prunes the hypothesis space before any
test run.

### Losing context across a