benchmark-context

# MemoryBench Integration Skill

Automatically benchmark your custom memory implementation against established systems like Supermemory, Mem0, and Zep.

## What This Skill Does

When you invoke this skill from your project, it handles the complete benchmarking process end-to-end:

1. **Asks you 5 questions** about your setup preferences
2. **Analyzes your memory code** to understand how it works
3. **Generates integration code** automatically
4. **Runs the full benchmark** comparing your system to competitors
5. **Shows you the results** with clear performance comparisons

No manual commands needed - everything runs automatically from start to finish.

## When to Use This Skill

Use this skill when you:
- Built a custom memory/context system and want to see how it performs
- Need objective metrics comparing your implementation to industry solutions
- Want to benchmark on standardized datasets (conversational memory, RAG, dialogue)
- Are considering which memory system to use and want data-driven comparison

## How It Works

### The 7 Automated Phases

**Phase 1: Setup**
- Uses the in-repository `./memorybench` checkout when present, or clones a pinned upstream revision into your project when bootstrapping a new integration
- Installs dependencies with bun
- Verifies environment is ready

**Phase 2: Discovery**
- Uses AI agents to analyze your memory codebase
- Identifies initialization, ingestion, and search methods
- Detects required configuration and API keys
- Understands your data formats

**Phase 3: Code Generation**
- Creates a provider adapter implementing the MemoryBench interface
- Copies and adapts your memory code into the provider
- Generates custom prompts if needed for your result format
- See [Provider Template Reference](references/provider-template.md)

**Phase 4: Registration**
- Updates `src/types/provider.ts` with your provider name
- Registers provider in `src/providers/index.ts`
- Adds configuration in `src/utils/config.ts`
- Documents environment variables

**Phase 5: Configuration**
- Creates `.env.local` with required API keys
- Asks for your provider credentials
- Asks for comparison provider keys (if selected)
- Asks for OpenAI/Anthropic key for judging

**Phase 6: Validation**
- Runs quick test with single question
- Verifies provider initialization works
- Confirms ingestion and search work correctly
- Asks if you want to continue or debug if issues arise

**Phase 7: Benchmark Execution**
- Runs full benchmark automatically
- Shows real-time progress (ingestion, indexing, search, answers, evaluation)
- Compares against selected competitors
- Presents final results with accuracy and latency metrics

## Initial Questions

The skill will ask you these questions upfront:

### 1. Provider Name
What should we call your memory provider?
- Use lowercase, no spaces (e.g., "mymemory", "contextengine")

### 2. Memory Code Location
Where is your memory implementation?
- Examples: `src/lib/memory`, `libs/memory`, `src/services/context`

### 3. Benchmark Dataset
Which dataset matches your use case?

- **LoCoMo** - Long-term conversational memory across multiple sessions spanning days/weeks
  - Best for: Chat apps, AI assistants, customer support bots

- **LongMemEval** - Memory with long documents and complex retrieval
  - Best for: RAG systems, document analysis, knowledge bases

- **ConvoMem** - Multi-turn conversation understanding and context tracking
  - Best for: Dialogue systems, interview bots, meeting assistants

See [Benchmarks Reference](references/benchmarks.md) for detailed information.

### 4. Comparison Targets (Multi-select)
Which systems to compare against?

- **Supermemory** - Fast hybrid vector + graph memory with automatic extraction
- **Mem0** - Persistent memory for AI agents with graph relationships
- **Zep** - Long-term conversation memory with automatic summarization
- **Filesystem** - Baseline vector search (no API required)
- **RAG** - Baseline RAG with LLM extraction (no API required)

### 5. Test Size
How many questions to benchmark?

- **Small** (5 questions) - Quick validation, ~2-5 minutes
- **Medium** (20 questions) - Good sample size, ~10-15 minutes
- **Full** (all questions) - Complete evaluation, ~30-60 minutes

## Working Directory Management

**Important:** You must run this skill from your project root, NOT from memorybench.

```
your-project/              ← Run skill from here
├── src/
│   └── lib/memory/        ← Your memory implementation
└── memorybench/           ← Skill clones this automatically
    └── src/providers/     ← Your provider adapter goes here
```

The skill will:
- Verify you're in your project (not in memorybench)
- Clone memorybench to `./memorybench`
- Use relative paths (`../src/lib/memory`) when analyzing your code
- Run benchmarks with `cd memorybench && bun run src/index.ts ...`

## What Gets Created

After the skill runs, you'll have:

```
your-project/
└── memorybench/
    ├── .env.local                              # Your API keys
    ├── src/
    │   ├── providers/
    │   │   └── {yourname}/
    │   │       ├── index.ts                    # Provider implementation
    │   │       └── prompts.ts                  # Custom prompts (optional)
    │   ├── types/provider.ts                   # Updated with your provider
    │   └── providers/index.ts                  # Registered
    └── data/runs/{run-id}/                     # Benchmark results
        ├── checkpoint.json                     # Run state
        ├── results/                            # Per-question results
        └── report.json                         # Final metrics
```

## After Completion

Once the benchmark finishes, the skill shows:

**Summary Scores:**
- Accuracy percentage for each provider
- Average search latency
- Success rate (questions answered vs failed)

**Key Findings:**
- "Your provider achieved 76% accuracy vs 82% for Supermemory"
- "Search latency: 145ms (yours) vs 98ms (best competitor)"

**Next Steps:**
- View detailed