release-it

# Release It! Framework

Framework for designing, deploying, and operating production-ready software. The software that passes QA is not the software that survives production — production is hostile, and systems must expect and handle failure at every level.

## Core Principle

**Every system will eventually be pushed beyond its design limits.** The question is not whether failures happen, but whether your system degrades gracefully or collapses catastrophically. Production-ready software is not just correct — it is resilient, observable, and operates through partial failures without human intervention.

## Scoring

**Goal: 10/10.** When reviewing or creating production systems, rate them 0-10 against the principles below — 10/10 means full alignment, lower scores indicate gaps. Always give the current score and the specific improvements needed to reach 10/10.

## The Release It! Framework

Six areas that determine whether software survives contact with production:

### 1. Stability Anti-Patterns

**Core concept:** Failures propagate through integration points and cascade across system boundaries. The most dangerous patterns are not bugs in your code — they are emergent behaviors when systems interact under stress.

**Why it works:** Every production outage traces back to one or more of these predictable, recurring patterns; recognizing them lets you eliminate the cracks before production traffic finds them.

**Key insights:**
- Integration points are the number-one killer — every socket, HTTP call, or queue is a risk
- Slow responses are worse than no response: they tie up threads, exhaust pools, and propagate delay up the call chain
- Unbounded result sets turn a harmless query into an out-of-memory crash once data outgrows test assumptions
- Users generate load no test predicts — bots, retry storms, flash crowds; self-denial attacks happen when your own marketing overwhelms your infrastructure
- Blocked threads are the silent killer — deadlocks and contention show no errors until everything stops

**Code applications:**

| Context | Pattern | Example |
|---------|---------|---------|
| HTTP calls | Assume every remote call can fail, hang, or return garbage | Wrap all external calls with timeout + circuit breaker |
| Database queries | Enforce result set limits | Add `LIMIT`; paginate all list endpoints |
| Thread pools | Isolate pools per dependency | Separate pool for payment gateway vs. search |
| Marketing events | Coordinate launches with capacity planning | Pre-scale before Black Friday; queue coupon redemptions |

See: [references/anti-patterns.md](references/anti-patterns.md) for each anti-pattern with failure scenarios and detection strategies.

### 2. Stability Patterns

**Core concept:** Counter each anti-pattern with a stability pattern: circuit breakers stop cascades, bulkheads isolate blast radius, timeouts reclaim stuck resources. Together they make a system bend under load instead of breaking.

**Why it works:** These patterns accept failure as inevitable and design the response to it — a circuit breaker that trips is the system working correctly, protecting itself from a downstream failure.

**Key insights:**
- Circuit Breaker: three states (closed, open, half-open) — trips after threshold failures, periodically tests recovery
- Timeouts: every outbound call needs connect AND read timeouts, propagated up the call chain
- Retry with exponential backoff + jitter prevents thundering herd on recovery
- Fail Fast: reject requests you know will fail instead of wasting resources; Handshaking lets the server decline work before it's sent
- Steady State: systems accumulate cruft (logs, sessions, temp files) — design automatic cleanup
- Let It Crash: a clean restart often beats limping along in an unknown state

**Code applications:**

| Context | Pattern | Example |
|---------|---------|---------|
| Service calls | Circuit Breaker | Open after 5 failures in 60s; half-open after 30s |
| Resource isolation | Bulkhead | Dedicated connection pools for critical vs. non-critical |
| Network calls | Timeout with propagation | Connect 1s, read 5s; propagate deadline downstream |
| Retries | Backoff + jitter + budget | Base 100ms, max 3 retries, 20% fleet retry budget |
| Data cleanup | Steady State | Purge sessions >24h; rotate logs at 500MB |

See: [references/stability-patterns.md](references/stability-patterns.md) for state machines, threshold tuning, and pattern combinations.

### 3. Capacity and Availability

**Core concept:** Capacity is not one number — it is a multi-dimensional function of CPU, memory, network, disk I/O, connection pools, and threads. Capacity planning means knowing which resource bottlenecks first, and at what load.

**Why it works:** Untested systems fail at peak load — the worst possible moment. Knowing actual (not theoretical) limits lets you set realistic SLAs and scale before users hit the wall.

**Key insights:**
- Test taxonomy: load test (expected traffic), stress test (beyond limits), soak test (sustained, catches leaks), spike test (sudden bursts)
- Universal Scalability Law: throughput never scales linearly — contention and coherence costs cause diminishing returns
- Pool exhaustion looks identical to a database outage from the application's perspective; size pools from measured concurrency, not defaults
- "The cloud is infinitely scalable" is a myth — auto-scaling has lag, cold starts, and hard limits

**Code applications:**

| Context | Pattern | Example |
|---------|---------|---------|
| Load testing | Ramp to peak, then 2x, observe degradation | Increase RPS until latency exceeds SLO |
| Connection pools | Size from measured concurrency | Set pool to P99 active connections + 20% headroom |
| Soak testing | 80% capacity for 24-72 hours | Catch memory/connection/file-handle leaks |
| Capacity model | Document bottleneck per service | "Service X is memory-bound at 2000 RPS; 4GB per instance" |

See: [references/capacity-planning.md](references/ca