chunking-strategy

# Chunking Strategy for RAG Systems

## Overview

Provides chunking strategies for RAG systems, vector databases, and document processing. Recommends chunk sizes, overlap percentages, and boundary detection methods; validates semantic coherence; evaluates retrieval metrics.

## When to Use

Use when building or optimizing RAG systems, vector search pipelines, document chunking workflows, or performance-tuning existing systems with poor retrieval quality.

## Instructions

### Choose Chunking Strategy

Select based on document type and use case:

1. **Fixed-Size Chunking** (Level 1)
   - Use for simple documents without clear structure
   - Start with 512 tokens and 10-20% overlap
   - Adjust: 256 for factoid queries, 1024 for analytical

2. **Recursive Character Chunking** (Level 2)
   - Use for documents with structural boundaries
   - Hierarchical separators: paragraphs → sentences → words
   - Customize for document types (HTML, Markdown, JSON)

3. **Structure-Aware Chunking** (Level 3)
   - Use for structured content (Markdown, code, tables, PDFs)
   - Preserve semantic units: functions, sections, table blocks
   - Validate structure preservation post-split

4. **Semantic Chunking** (Level 4)
   - Use for complex documents with thematic shifts
   - Embedding-based boundary detection with 0.8 similarity threshold
   - Buffer size: 3-5 sentences

5. **Advanced Methods** (Level 5)
   - Late Chunking for long-context models
   - Contextual Retrieval for high-precision requirements
   - Monitor computational cost vs. retrieval gain

Reference: [references/strategies.md](references/strategies.md).

### Implement Chunking Pipeline

1. **Pre-process documents**
   - Analyze structure, content types, information density
   - Identify multi-modal content (tables, images, code)

2. **Select parameters**
   - Chunk size: embedding model context window / 4
   - Overlap: 10-20% for most cases
   - Strategy-specific settings

3. **Process and validate**
   - Apply chunking strategy
   - Validate coherence: run `evaluate_chunks.py --coherence` (see below)
   - Test with representative documents

4. **Evaluate and iterate**
   - Measure precision and recall
   - If precision < 0.7: reduce chunk_size by 25% and re-evaluate
   - If recall < 0.6: increase overlap by 10% and re-evaluate
   - Monitor latency and memory usage

Reference: [references/implementation.md](references/implementation.md).

### Validate Chunk Quality

Run validation commands to assess chunk quality:

```bash
# Check semantic coherence (requires sentence-transformers)
python -c "
from sentence_transformers import SentenceTransformer
model = SentenceTransformer('all-MiniLM-L6-v2')
chunks = [...]  # your chunks
embeddings = model.encode(chunks)
similarity = (embeddings @ embeddings.T).mean()
print(f'Cohesion: {similarity:.3f}')  # target: 0.3-0.7
"

# Measure retrieval precision
python -c "
relevant = sum(1 for c in retrieved if c in relevant_chunks)
precision = relevant / len(retrieved)
print(f'Precision: {precision:.2f}')  # target: >= 0.7
"

# Check chunk size distribution
python -c "
import numpy as np
sizes = [len(c.split()) for c in chunks]
print(f'Mean: {np.mean(sizes):.0f}, Std: {np.std(sizes):.0f}')
print(f'Min: {min(sizes)}, Max: {max(sizes)}')
"
```

Reference: [references/evaluation.md](references/evaluation.md).

## Examples

### Fixed-Size Chunking

```python
from langchain.text_splitter import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
    chunk_size=256,
    chunk_overlap=25,
    length_function=len
)
chunks = splitter.split_documents(documents)
```

### Structure-Aware Code Chunking

```python
import ast

def chunk_python_code(code):
    tree = ast.parse(code)
    chunks = []
    for node in ast.walk(tree):
        if isinstance(node, (ast.FunctionDef, ast.ClassDef)):
            chunks.append(ast.get_source_segment(code, node))
    return chunks
```

### Semantic Chunking

```python
def semantic_chunk(text, similarity_threshold=0.8):
    sentences = split_into_sentences(text)
    embeddings = generate_embeddings(sentences)
    chunks, current = [], [sentences[0]]
    for i in range(1, len(sentences)):
        sim = cosine_similarity(embeddings[i-1], embeddings[i])
        if sim < similarity_threshold:
            chunks.append(" ".join(current))
            current = [sentences[i]]
        else:
            current.append(sentences[i])
    chunks.append(" ".join(current))
    return chunks
```

## Best Practices

### Core Principles
- Balance context preservation with retrieval precision
- Maintain semantic coherence within chunks
- Optimize for embedding model context window constraints

### Implementation
- Start with fixed-size (512 tokens, 15% overlap)
- Iterate based on document characteristics
- Test with domain-specific documents before deployment

### Pitfalls to Avoid
- Over-chunking: context-poor small chunks
- Under-chunking: missing information in oversized chunks
- Ignoring semantic boundaries and document structure
- One-size-fits-all for diverse content types

## Constraints and Warnings

### Resource Considerations
- Semantic methods require significant compute resources
- Late chunking needs long-context embedding models
- Complex strategies increase processing latency
- Monitor memory for large document batches

### Quality Requirements
- Validate semantic coherence post-processing
- Test with representative documents before deployment
- Ensure chunks maintain standalone meaning
- Implement error handling for malformed content

## References

- [strategies.md](references/strategies.md) - Detailed strategies
- [implementation.md](references/implementation.md) - Implementation guidelines
- [evaluation.md](references/evaluation.md) - Performance metrics
- [tools.md](references/tools.md) - Libraries and frameworks
- [research.md](references/research.md) - Research papers
- [advanced-strategies.md](references/advanced-strategies.md) - 11 advanced methods
- [semantic-met