Skill1.8k repo starsupdated 5d ago

eval-harness

The eval-harness skill provides structured frameworks for systematically testing agent and skill performance through benchmarking, quality scoring, and regression testing. Use it after creating or modifying agents and skills, during periodic quality audits, and before promoting skills to production to ensure outputs meet accuracy, completeness, and consistency thresholds.

View source Repository: babysitter

Install in Claude Code

Copy

git clone --depth 1 https://github.com/a5c-ai/babysitter /tmp/eval-harness && cp -r /tmp/eval-harness/library/methodologies/everything-claude-code/skills/eval-harness ~/.claude/skills/eval-harness

Then start a new Claude Code session; the skill loads automatically.

Definition

SKILL.md

- Define test cases with known-correct outputs
- Run agent against each test case
- Score: accuracy, completeness, relevance
- Compare against baseline performance
- Track performance over time

### 2. Skill Quality Testing
- Verify skill instructions produce expected outcomes
- Test edge cases and boundary conditions
- Measure consistency across multiple runs
- Check for harmful or incorrect outputs
- Validate against ground truth

### 3. Regression Suite
- Collection of previously-passing test cases
- Run after any agent/skill modification
- Flag regressions with before/after comparison
- Maintain pass rate threshold (>= 95%)

### 4. Process Verification
- End-to-end process execution with known inputs
- Verify each phase produces expected outputs
- Check task ordering and dependency satisfaction
- Measure total execution time

## Quality Scoring

### Accuracy Score (0-100)
- Correctness of output vs expected
- Partial credit for partially correct outputs
- Penalty for hallucinated or fabricated content

### Completeness Score (0-100)
- Coverage of required output elements
- Missing sections flagged and scored
- Bonus for useful additional context

### Consistency Score (0-100)
- Run same input 3 times
- Compare outputs for semantic similarity
- Flag inconsistencies

### Composite Score
- (accuracy * 0.4 + completeness * 0.3 + consistency * 0.3)
- Threshold: 80 to pass

## When to Use

- After creating new agents or skills
- After modifying existing agents or skills
- Periodic quality audits
- Before promoting skills to production

## Agents Used

- Used by process-level evaluation orchestrators
- No specific agent dependency (evaluates other agents)