measure-instrumentation-spec

<!-- PM-Skills | https://github.com/product-on-purpose/pm-skills | Apache 2.0 -->
# Instrumentation Spec

An instrumentation spec defines what analytics events to track, when to fire them, and what properties to include. It serves as a contract between product and engineering, ensuring consistent data collection that enables accurate measurement. Good instrumentation specs prevent the "we can't answer that question because we didn't track it" problem.

## When to Use

- Before engineering implements a new feature
- When defining analytics requirements for experiments
- When auditing existing tracking for gaps or inconsistencies
- When onboarding a new analytics tool
- Before launch to ensure measurement is in place

## When NOT to Use

- You are specifying the dashboard built on top of the events -> use `measure-dashboard-requirements`
- You need experiment-specific metrics and variants, not product-wide tracking -> use `measure-experiment-design`
- The feature itself is not yet specified (no flows to instrument) -> use `deliver-prd` first
- You are analyzing data you already collect -> use `measure-experiment-results` or `measure-survey-analysis`

## Instructions

When asked to create an instrumentation spec, follow these steps:

1. **Define Analytics Goals**
   Start with the questions you need to answer. What will you measure? What decisions will this data inform? This prevents over-instrumentation while ensuring nothing important is missed.

2. **Identify Events to Track**
   List each user action or system event that should be tracked. Follow consistent naming conventions (typically `noun_verb` or `verb_noun` in snake_case). Each event should represent a distinct, meaningful action.

3. **Specify Event Triggers**
   For each event, describe exactly when it fires. Be precise: "When user clicks Submit button" vs. "When form is submitted successfully." These are different events with different meanings.

4. **Define Event Properties**
   List the properties (attributes) attached to each event. Include property name, data type, description, and example values. Properties provide context that makes events useful.

5. **Document User Properties**
   Identify persistent user-level attributes that should be associated with all events (e.g., subscription tier, account creation date). These enable segmentation in analysis.

6. **Address PII and Privacy**
   Flag any properties that contain personally identifiable information. Document how PII should be handled - hashing, encryption, or exclusion.

   *When any model input, output, retrieval context, or tool call is captured*, extend this section to cover the trace as well. The trigger is capture, not authorship: it fires whether the sensitive text was typed by a user, retrieved from a tenant's documents, carried in a system prompt, passed to or returned from a tool, read out of an uploaded file or a batch job, or generated by the model itself. A feature with no direct user input can still write a customer's contract text into a trace store.

   A trace is not an event: an event carries properties you chose in advance, while a trace carries free text that can contain anything, including data no property schema anticipated. Decide and record:

   - **Data classes captured.** Name them (user text, retrieved documents, system prompt, tool arguments and results, file contents, model output). "The trace" is not an answer.
   - **Minimization, at both boundaries.** What is dropped or redacted *before the trace leaves the process*, and separately what is dropped *before it is stored*. A redactor that runs only at the storage layer has already sent the raw text over the wire.
   - **Access.** Who can read traces, and whether reads are logged. Trace stores are routinely the least-governed copy of the most sensitive data a feature handles.
   - **Retention and deletion.** How long, what deletes them, and how a deletion request reaches a trace that was copied into an evaluation set.
   - **Consent and opt-out.** Whether the subject agreed, and what the feature does when they decline.
   - **Sampling.** What fraction is captured and how that sample is chosen. A uniform sample is the wrong instrument for finding rare failures; if traces exist to diagnose bad output, oversample the flagged cases and say so.

7. **Create Testing Checklist**
   Define how QA should verify that tracking is implemented correctly. Include steps to validate events fire at the right times with correct properties.

## Output Format

Use the template in `references/TEMPLATE.md` to structure the output. A complete spec fills every template section: Overview; Event Inventory; User Properties; PII & Privacy Considerations; Implementation Notes; and Testing Checklist.

## Quality Checklist

Before finalizing, verify:

- [ ] Event names follow consistent naming convention
- [ ] Each event has a clear, unambiguous trigger
- [ ] Properties include data types and example values
- [ ] PII is identified and handling is documented
- [ ] Events map to the analytics questions you need to answer
- [ ] Testing checklist enables QA verification
- [ ] If any model input, output, retrieval context, or tool call is captured: the data classes are named, minimization is decided at both the egress and the storage boundary, access and read-logging, retention and deletion, consent, and sampling are all decided, not deferred

## Examples

See `references/EXAMPLE.md` for a completed example.