extracting-keywords

# Extracting Keywords

Extract keywords from text using YAKE (Yet Another Keyword Extractor), an unsupervised statistical keyword extraction algorithm.

## Installation

**First time only:** Install YAKE with optimized dependencies to avoid unnecessary downloads.

```bash
cd /home/claude
uv venv yake-venv --system-site-packages
uv pip install yake --python yake-venv/bin/python --no-deps
uv pip install jellyfish segtok regex --python yake-venv/bin/python
```

This reuses system packages (numpy, networkx) instead of downloading them (~0.08s vs ~5s).

## Stopwords Configuration

**Built-in YAKE stopwords (34 languages):** Use `lan="<code>"` parameter
- See Parameters section below for all 34 supported language codes
- English (`lan="en"`) is the default

**Custom domain stopwords (bundled in `assets/`):**

**AI/ML:** `stopwords_ai.txt`
- English stopwords + 783 AI/ML domain-specific terms (1357 total)
- Filters AI/ML methodology noise (model, training, network, algorithm, parameter)
- Filters ML boilerplate (dataset, baseline, benchmark, experiment, evaluation)
- Filters technical terms (transformer, embedding, attention, optimization, inference)
- Includes full lemmatization (train/trains/trained/training/trainer)
- Use for AI/ML papers, technical reports, machine learning literature
- **Performance impact:** +4-5% runtime vs English stopwords

**Life Sciences:** `stopwords_ls.txt`
- English stopwords + 719 life sciences domain-specific terms (1293 total)
- Filters research methodology noise (study, results, analysis, significant, observed)
- Filters academic boilerplate (paper, manuscript, publication, review, editing)
- Filters statistical terms (correlation, distribution, deviation, variance)
- Filters clinical terms (patient, treatment, diagnosis, symptom, therapy)
- Filters biology/medicine (cell, tissue, protein, gene, organism)
- Includes full lemmatization (analyze/analyzes/analyzed/analyzing/analysis)
- Use for biomedical papers, clinical studies, research articles, scientific literature
- **Performance impact:** +4-5% runtime vs English stopwords

## Basic Usage

```python
import yake

# Read text
with open('document.txt', 'r') as f:
    text = f.read()

# Extract with English stopwords (default)
kw_extractor = yake.KeywordExtractor(
    lan="en",           # Language code
    n=3,                # Max n-gram size (1-3 word phrases)
    dedupLim=0.9,       # Deduplication threshold (0-1)
    top=20              # Number of keywords to return
)

keywords = kw_extractor.extract_keywords(text)

# Display results (lower score = more important)
for kw, score in keywords:
    print(f"{score:.4f}  {kw}")
```

## Domain-Specific Extraction

### Using Life Sciences Stopwords

**Option 1: Install custom stopwords file**

```bash
# Copy life sciences stopwords to YAKE package
cp assets/stopwords_ls.txt /home/claude/yake-venv/lib/python3.12/site-packages/yake/core/StopwordsList/stopwords_ls.txt

# Use with lan="ls"
kw_extractor = yake.KeywordExtractor(lan="ls", n=3, top=20)
```

**Option 2: Load custom stopwords directly**

```python
# Load stopwords from file
with open('assets/stopwords_ls.txt', 'r') as f:
    custom_stops = set(line.strip().lower() for line in f)

# Pass to extractor
kw_extractor = yake.KeywordExtractor(
    stopwords=custom_stops,
    n=3,
    top=20
)
```

### Using AI/ML Stopwords

```python
# Load AI/ML stopwords
with open('/mnt/skills/user/extracting-keywords/assets/stopwords_ai.txt', 'r') as f:
    ai_stops = set(line.strip().lower() for line in f)

# Extract with AI stopwords
kw_extractor = yake.KeywordExtractor(
    stopwords=ai_stops,
    n=3,
    top=20
)
keywords = kw_extractor.extract_keywords(text)
```

## Deeper Extraction (n=2 + n=3 Combined)

For more comprehensive extraction, run both n=2 and n=3 and consolidate results. This captures both focused phrases and broader context with ~100% time overhead (still <2s for large documents).

```python
import yake

# Load domain stopwords
with open('/mnt/skills/user/extracting-keywords/assets/stopwords_ai.txt', 'r') as f:
    stops = set(line.strip().lower() for line in f)

# Extract with n=2 (captures focused phrases)
kw_n2 = yake.KeywordExtractor(stopwords=stops, n=2, dedupLim=0.9, top=50)
results_n2 = kw_n2.extract_keywords(text)

# Extract with n=3 (captures broader context)
kw_n3 = yake.KeywordExtractor(stopwords=stops, n=3, dedupLim=0.9, top=50)
results_n3 = kw_n3.extract_keywords(text)

# Consolidate: union with score averaging for overlaps
combined = {}
for kw, score in results_n2:
    combined[kw] = score
for kw, score in results_n3:
    if kw in combined:
        combined[kw] = (combined[kw] + score) / 2
    else:
        combined[kw] = score

# Sort by score (lower = more important)
consolidated = sorted(combined.items(), key=lambda x: x[1])

# Display top 30
for kw, score in consolidated[:30]:
    print(f"{score:.4f}  {kw}")
```

**Benefits:**
- n=2 extracts cleaner domain-specific phrases ("disk move", "error rate")
- n=3 captures contextual combinations ("Move disk 1", "per-step error rate")
- Consolidation provides richer keyword set for topic modeling or search indexing

**Performance:**
- Combined approach: ~2x runtime of single extraction
- Typical timing: 0.4s (small doc) to 1.0s (large doc)
- Use when quality matters more than speed

## Parameters

**lan** (str): Language code for built-in stopwords
- `"en"` - English (default)
- `"ai"` - AI/ML (if stopwords_ai.txt installed in YAKE)
- `"ls"` - Life sciences (if stopwords_ls.txt installed in YAKE)

**Built-in YAKE languages (34 total):**
- `"ar"` - Arabic
- `"bg"` - Bulgarian  
- `"br"` - Breton
- `"cz"` - Czech
- `"da"` - Danish
- `"de"` - German
- `"el"` - Greek
- `"es"` - Spanish
- `"et"` - Estonian
- `"fa"` - Farsi/Persian
- `"fi"` - Finnish
- `"fr"` - French
- `"hi"` - Hindi
- `"hr"` - Croatian
- `"hu"` - Hungarian
- `"hy"` - Armenian
- `"id"` - Indonesian
- `"it"` - Italian
- `"ja"` - Japanese
- `"lt"` - Lithuanian
- `"lv"` - Latvian
- `"n