overfitting-audit-mcp

# overfitting-audit-mcp

[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
[![Live on MCPize](https://img.shields.io/badge/Live%20on-MCPize-6d28d9)](https://mcpize.com/mcp/overfitting-audit-mcp)

An MCP server that answers "is this edge real, or a testing-hundreds-of-variants artifact?" — implementing the Probability of Backtest Overfitting (CSCV method), Deflated Sharpe Ratio, Minimum Backtest Length, and purged/embargoed cross-validation splits.

## The problem this solves

Testing enough parameter combinations against the same historical data will eventually produce a great-looking backtest by chance alone. Standard backtest metrics (Sharpe, win rate, profit factor) don't distinguish a genuine edge from the best-looking result out of hundreds of near-identical variants. This audits for that specific failure mode directly, rather than trusting a single strong-looking curve.

## Tools

### `probability_of_backtest_overfitting`
Combinatorially Symmetric Cross-Validation (CSCV) method — estimates the probability that a strategy's in-sample performance rank won't hold out-of-sample.

### `deflated_sharpe_ratio`
Adjusts a Sharpe ratio for the number of trials run and the non-normality of returns, so it can't be inflated just by testing more variants.

### `minimum_backtest_length`
The minimum number of independent trials/observations needed before a given Sharpe ratio is statistically meaningful at all.

### `purged_cv_split`
Generates purged and embargoed cross-validation splits for time-series backtests, preventing the lookahead leakage that ordinary k-fold CV introduces on financial data.

## Use it

**Hosted (recommended):** [MCPize](https://mcpize.com/mcp/overfitting-audit-mcp) — free tier, paid Pro tier for higher limits.

**Self-host:**
```bash
npm install
node server.js
```

## Part of the AlgoForge suite

Prop-firm and quant-validation tools for algo traders: [prop-rules-mcp](https://github.com/tylerscomic-lab/prop-rules-mcp), [trade-journal-mcp](https://github.com/tylerscomic-lab/trade-journal-mcp), [payout-calc-mcp](https://github.com/tylerscomic-lab/payout-calc-mcp), [econ-calendar-mcp](https://github.com/tylerscomic-lab/econ-calendar-mcp), [montecarlo-validator-mcp](https://github.com/tylerscomic-lab/montecarlo-validator-mcp), [walkforward-validator-mcp](https://github.com/tylerscomic-lab/walkforward-validator-mcp), [pinescript-audit-mcp](https://github.com/tylerscomic-lab/pinescript-audit-mcp), [backtest-cost-sensitivity-mcp](https://github.com/tylerscomic-lab/backtest-cost-sensitivity-mcp), [pinescript-mcp](https://github.com/tylerscomic-lab/pinescript-mcp).

## License

MIT