Tokenectomy

# Tokenectomy Razor

> **Fast, deterministic log surgery and secret redaction for AI coding agents — purge 90%+ framework noise, redact credentials with O(N) ReDoS immunity, sub-millisecond latency. Written in safe Rust.**

*Tokenectomy (noun): **token** + **-ectomy** (surgical removal) — the precise excision of wasteful tokens from LLM context windows.*

### High-Performance Log Surgery & Secret Redaction Engine for AI Coding Agents

<p align="left">
  <a href="https://tokenectomy-web.vercel.app"><img src="https://img.shields.io/badge/Website-tokenectomy--web.vercel.app-000000?style=flat&logo=vercel" alt="Tokenectomy Razor official website" /></a>
  <a href="https://crates.io/crates/tokenectomy"><img src="https://img.shields.io/crates/v/tokenectomy.svg?logo=rust" alt="Tokenectomy Razor crate version on crates.io" /></a>
  <a href="https://www.npmjs.com/package/tokenectomy-razor"><img src="https://img.shields.io/npm/v/tokenectomy-razor.svg?logo=npm" alt="Tokenectomy Razor npm package version" /></a>
  <a href="https://github.com/Tokenectomy-Labs/Tokenectomy/actions/workflows/ci.yml"><img src="https://github.com/Tokenectomy-Labs/Tokenectomy/actions/workflows/ci.yml/badge.svg" alt="Tokenectomy Razor CI build status" /></a>
  <a href="SECURITY.md"><img src="https://img.shields.io/badge/Security-Audited%20(RustSec)-2ea44f?logo=rust" alt="Tokenectomy Razor RustSec security audit status" /></a>
  <a href="LICENSE"><img src="https://img.shields.io/badge/License-MIT-blue.svg" alt="Tokenectomy Razor MIT License" /></a>
  <a href="https://glama.ai/mcp/servers/Tokenectomy-Labs/Tokenectomy"><img src="https://img.shields.io/badge/Glama.ai-Tokenectomy--Razor-purple" alt="Tokenectomy Razor on Glama.ai" /></a>
  <a href="https://mcpservers.org/servers/tokenectomy-labs/tokenectomy"><img src="https://mcpservers.org/badge.svg" alt="Listed on mcpservers.org" /></a>
  <a href="https://registry.modelcontextprotocol.io"><img src="https://img.shields.io/badge/Official%20MCP%20Registry-io.github.Tokenectomy--Labs%2Frazor-brightgreen" alt="Official MCP Registry" /></a>
  <a href="https://github.com/marketplace/actions/tokenectomy-razor"><img src="https://img.shields.io/badge/GitHub%20Marketplace-Tokenectomy%20Razor-blue?logo=githubactions" alt="Tokenectomy Razor GitHub Actions Marketplace" /></a>
  <a href="https://tokenectomy-labs.github.io/Tokenectomy"><img src="https://img.shields.io/badge/Docs-GitHub%20Pages-blue?logo=googledocs" alt="Tokenectomy Razor documentation site" /></a>
</p>

- **Official MCP Registry:** `mcp-name: io.github.Tokenectomy-Labs/razor`

---

## 📋 Table of Contents

- [What It Does](#what-it-does)
- [Key Features](#technical-highlights)
- [Benchmarks](#verifiable-benchmarks)
- [Installation](#installation)
- [MCP Integration](#model-context-protocol-mcp-integration)
- [Usage by Use Case](#quick-start)
- [Advanced Features](#advanced-usage)
- [Comparison](#how-tokenectomy-compares)
- [FAQ](#frequently-asked-questions)
- [Roadmap](#roadmap)
- [Security](#security--reliability-invariants)
- [Contributing](#contributing)

---

## What It Does

Tokenectomy Razor is an autonomous, machine-to-machine (M2M) Model Context Protocol (MCP) server and stream processing engine written in safe Rust. It intercepts error logs from AI agents, strips 90%+ of framework noise, automatically redacts secrets (JWTs, API keys, database credentials), and caches sanitized contexts with a 24-hour TTL—all without sending raw data to external services.

### In 30 Seconds

**The Problem:**
- AI agents waste tokens on framework noise (`node_modules`, `site-packages`, `.cargo/registry`)
- Sensitive credentials accidentally leak into LLM logs (AWS keys, database URLs, API tokens)
- Repeated identical errors cost money for every retry

**The Solution:**
```
Raw Error Log (38K tokens + secrets)
    ↓
[Redact secrets locally] → [Filter framework frames] → [Extract user code]
    ↓
Sanitized Context (2K tokens, no secrets) → Safe to send to LLM
```

### Real Example

**Before:**
```bash
$ cat error.log | head -20
Error in /home/user/.cargo/registry/src-xxx/tokio-1.35/src/runtime/mod.rs:12345
  at /home/user/.cargo/registry/src-yyy/serde/src/lib.rs:456
  Database connection failed: postgresql://admin:secretpass@db.example.com:5432/mydb
  JWT Auth token: eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIn0...
  [... 500+ more framework frames ...]
```

**After:**
```bash
$ cat error.log | razor --scrub
Error in /home/user/src/main.rs:42
  at /home/user/src/utils.rs:18
  Database connection failed: [CONNECTION_STRING_REDACTED]
  JWT Auth token: [JWT_REDACTED]
```

**Benefits:**
- ✅ 95% smaller context (2K vs 38K tokens) → Save money on LLM API calls
- ✅ Zero secrets in logs → Sleep better at night
- ✅ Identical errors cached → Second retry costs $0

---

## Technical Highlights

```
                  ┌──────────────────────────────────────────────┐
   Agent Error     │              TOKENECTOMY RAZOR               │    Sanitized Context
   Dump (38K toks) │  - Polyglot Stack Frame Filter               │ ──►  (2K toks) ──► LLM
  ────────────────►│  - Deterministic Secret Redactor (O(N))      │
                   │  - SHA-256 Idempotency Cache (24h TTL)       │
                   └──────────────────────────────────────────────┘
```

- **Deep Polyglot Trace Surgery**: In-memory parsing across Rust, Python, TypeScript/JavaScript, Go, Java/Kotlin (Spring Boot 3, Tomcat, Hibernate, Netty, Undertow, HikariCP), C/C++ (AddressSanitizer, GDB, glibc), and PHP. Surgically filters noisy framework internals and runtime boilerplate while isolating genuine user application code frames.
- **Static AST Code Analysis Engine (`analyze_code`)**: High-throughput static AST analysis detecting unclosed handles, resource leaks, and security vulnerabilities with bounded execution limits (50K AST nodes, 10 MB file limit) and precise LSP UTF-16 coordinates.
- **Glama Grade A TDQS Compliance**: 100% Tool Definition Quality Score with explicit schema boundaries, runtime preconditions, and full disclosure across all MCP tools.
- **AI Gateway Reverse Proxy (`--proxy`)**: Transparently intercepts prompt streams on `127.0.0.1:8080`, performing real-time token excision and credential sanitization before upstream forwarding to OpenAI, Anthropic, or Ollama.
- **Zero-Knowledge Secret Redaction**: Linear-time deterministic regex engine strips JWTs, API tokens, cloud access keys, connection strings, and private keys prior to network transmission. All processing happens locally.
- **SHA-256 Idempotency Cache**: Stores deterministic responses with a 24-hour TTL. Repeated CI/CD or agent loop failures incur zero upstream API cost.
- **Path Traversal Containment**: All MCP filesystem access is canonicalized and locked to the workspace root boundary (`CWD`). No `../` escapes or symlink breakouts.
- **M2M Protocol Compliance**: Native JSON-RPC 2.0 stdio server compliant with the official Model Context Protocol specification.

---

## Verifiable Benchmarks

Performance metrics are hardware-grounded and reproducible via standalone benchmark suites:

| Benchmark Target | Workload Under Test | Verified Measurement | Result |
|---|---|---|:---:|
| **High-Volume Log Redaction** | 250,000 lines (24.44 MB) enterprise dump containing API keys and connection URIs | **333.49 ms (73.3 MB/sec, 749,652 lines/sec)** | Pass |
| **ReDoS Resistance** | 50,000-character pathological backtracking string | **1.44 ms** (Linear $O(N)$ evaluation) | Pass |
| **Thread Concurrency** | 100 concurrent OS threads executing simultaneous redaction and extraction | **100/100 completed in 27.35 ms (7,312 ops/sec)** | Pass |
| **Kernel Memory Footprint** | Peak Resident Memory during 250,000-line continuous stress test | **76.24 MB VmRSS** via `/proc/self/status` | Pass |

### Understanding the Benchmarks

| Metric | Why It Matters | What To Expect |
|--------|---|---|
| **73.3 MB/sec redaction throughput** | Most logs are <5MB; you'll redact them in milliseconds | <10ms for typical CI logs |
| **1.44ms ReDoS immunity** | Prevents malicious log payloads from DoS'ing your system | Safe to use in production with untrusted input |
| **76.24 MB peak memory** | Suitable for constrained CI/CD runners (GitHub Actions, GitLab) | Fits within 256MB limits comfortably |
| **7,312 ops/sec concurrent** | Multiple AI agents querying simultaneously | 100 concurrent requests handled safely |

> **Reproduce locally:**
> ```bash
> cargo test --release --test stress_benchmark -- --nocapture
> ```

---

## Installation

### Method 1: Instant via npx (Recommended for MCP Clients)

No Rust toolchain, native compilation, or manual path setup required:

```bash
npx -y tokenectomy-razor --mcp
```

Or install globally via npm:

```bash
npm install -g tokenectomy-razor
```

### Method 2: Cargo (crates.io)

```bash
cargo install tokenectomy
```

### Method 3: Precompiled Native Binaries (GitHub Releases)

Download zero-dependency, precompiled standalone binaries directly from [GitHub Releases](https://github.com/Tokenectomy-Labs/Tokenectomy/releases):

- **Linux:** `tokenectomy-linux-x86_64` (glibc), `tokenectomy-linux-x86_64-musl`, `tokenectomy-linux-aarch64`
- **macOS:** `tokenectomy-darwin-arm64` (Apple Silicon M1/M2/M3/M4), `tokenectomy-darwin-x86_64` (Intel)
- **Windows:** `tokenectomy-windows-x86_64.exe`

### Method 4: Build from Source

```bash
git clone https://github.com/Tokenectomy-Labs/Tokenectomy.git
cd Tokenectomy
cargo build --release
sudo cp target/release/razor /usr/local/bin/razor
```

### Method 5: Multi-Arch Container (GHCR)

```bash
docker pull ghcr.io/tokenectomy-labs/razor:latest
docker run -it ghcr.io/tokenectomy-labs/razor:latest --help
```

---

## Model Context Protocol (MCP) Integration

Configure Tokenectomy Razor as an autonomous background server across major AI agent environments:

### Claude Desktop

Add to `~/Library/Application Support/Claude/claude_desktop_config.json` (macOS) or `%APPDATA%\Claude\claude_deskto