ai-eval-plan
Design an evaluation plan for an LLM or AI feature before shipping it. Use when asked how to evaluate a prompt/model/agent, set up an eval harness, define quality metrics for an AI feature, or build a regression gate. Produces an eval plan — task definition, datasets, metrics & rubrics, baselines, automated + human evals, a pass bar, and a regression gate.
git clone --depth 1 https://github.com/mohitagw15856/pm-claude-skills /tmp/ai-eval-plan && cp -r /tmp/ai-eval-plan/exports/openclaw/ai-eval-plan ~/.claude/skills/ai-eval-planSKILL.md
# AI Eval Plan Skill You can't improve an AI feature you can't measure, and "it looks good in the demo" is not measurement. This skill produces an evaluation plan that turns a fuzzy quality goal into a repeatable, gated test — so a prompt change that quietly makes outputs worse can't ship. ## Required Inputs Ask for these only if they aren't already provided: - **The feature & task** — what the model does and what "good output" means to a user. - **Failure modes that matter** — what bad looks like (hallucination, wrong format, unsafe, off-tone, too slow). - **Available data** — any real examples, logs, or labelled cases; or note there are none yet. - **Who judges quality** — automated checks, an LLM judge, human raters, or a mix. - **The decision this gates** — ship/no-ship, model selection, or prompt iteration. ## Output Format ### Eval Plan: [feature] **1. What we're measuring** — the task, and a one-line definition of a good vs. bad response. **2. Eval dataset** - **Cases:** how many, where they come from (real logs > synthetic), and how they're split (smoke set vs. full set). - **Coverage:** the slices/scenarios that must be represented (edge cases, adversarial, each major input type). - **Golden answers / references:** present or not, and how they were created. **3. Metrics & rubric** - **Per-dimension scores** — define each dimension (e.g. correctness, grounding, format, safety, tone) on an explicit 1–5 rubric with anchor descriptions, not vibes. - **Automated checks** — deterministic assertions first (valid JSON, contains required fields, no PII, latency budget). - **LLM-as-judge** — the judge prompt, the rubric it applies, and how you guard against its bias (calibrate against human labels on a sample). - **Human eval** — when it's required (safety, subjective quality) and the rater instructions. **4. Baselines** — what each candidate is compared against (current prompt, previous model, a plain-prompt control). **5. The bar** — the explicit threshold to ship (e.g. "≥4.2 avg correctness, 0 safety failures, p95 < 3s") and what happens if it's missed. **6. Regression gate** — how this runs in CI on every change, and the score-drop threshold that blocks a merge. ## Quality Checks - [ ] Each metric has an explicit rubric with anchors — not just a name - [ ] Deterministic/automated checks are used wherever possible before reaching for an LLM judge - [ ] The LLM judge is calibrated against human labels on at least a sample - [ ] The eval set includes adversarial and edge cases, not just happy-path examples - [ ] There is a single, explicit numeric bar for the ship decision - [ ] The plan specifies how it runs as a regression gate, not just a one-time check ## Anti-Patterns - [ ] Do not rely on a single overall score — a feature can pass on average while failing every safety case - [ ] Do not trust an LLM judge you haven't calibrated against humans — it has its own blind spots and biases - [ ] Do not eval only on happy-path inputs — the failures live in the edges and the adversarial cases - [ ] Do not let the eval set leak into the prompt/few-shot examples — that's training on the test set - [ ] Do not define the pass bar after seeing the scores — set the threshold before you run, or it means nothing ## Based On LLM evaluation practice — task-grounded rubrics, LLM-as-judge with human calibration, and regression-gated CI evals.
Design a 360-degree feedback survey or write a structured 360 feedback report. Use when asked to build a 360 feedback process, write 360 feedback for a colleague, design a feedback survey, or produce a feedback report. Produces either a complete survey instrument with rating scales and open-ended questions, or a structured narrative feedback report with themes, strengths, and development areas.
Decode a 401k or workplace retirement plan — the real cost of its funds, the match's fine print, vesting math, and the plan features worth using or avoiding. Use when someone asks 'is my 401k any good', 'decode my 401k plan', 'which funds should I look at', or 'what fees am I paying'. Produces a fee decode in dollars-over-time, match and vesting math, a fund-lineup triage by cost, and the questions for HR or the plan administrator.
Design statistically rigorous A/B tests for product features, UI changes, onboarding flows, and pricing experiments. Use when asked to set up an experiment, design an A/B test, calculate sample size, or interpret test results. Produces a complete test plan with hypothesis, variant definitions, sample size, duration estimate, guardrail metrics, and a results interpretation guide.
Analyse a finished A/B test and write the readout — the result, whether it's statistically and practically significant, what it means, and the ship/no-ship call. Use when asked to analyse experiment results, write an A/B test readout, interpret test data, or decide whether to ship a variant. Produces a clear verdict with the lift and confidence, segment cuts, the risks (peeking, novelty, sample), and a recommendation. Distinct from planning a test — this reads results.
Generate a WCAG 2.2 accessibility audit checklist and remediation suggestions for any UI or design. Use when asked to audit for accessibility, check WCAG compliance, review a design for a11y issues, or create an accessibility remediation plan. Produces a prioritised checklist with pass/fail assessments and specific fixes.
Plan a trip that actually works with a disability or access need — confirm real accessibility (not just 'accessible' labels), book the assistance in advance, plan for equipment and medication, and build in the contingencies for when access breaks down. Use when someone says 'plan an accessible trip', 'travelling with a wheelchair/disability', 'book assistance for my flight', or 'will this hotel actually work for me'. Produces an access-verified itinerary, an assistance-booking checklist, an equipment/medication plan, and contingency scripts. Verify specifics with providers.
Request a reasonable accommodation at work or in education — frame it around the barrier and the adjustment (not your diagnosis), cite the right process, and navigate the back-and-forth constructively. Use when someone says 'I need a workplace accommodation', 'request reasonable adjustments', 'ADA/Equality Act accommodation', or 'how do I ask for accommodations for my disability/condition'. Produces the request letter, a barriers-and-adjustments map, disclosure guidance, and a plan for the interactive process. Not legal advice — routes to the formal process and to advocacy where needed.
Build a structured account plan for any key customer or target account. Use when asked to create an account plan, key account strategy, strategic account review, or territory plan. Produces a complete account plan with relationship map, growth opportunities, risks, and 90-day action plan.