pdf

# PDF Processing Guide

## Overview

This guide covers essential PDF processing operations using Python libraries and command-line tools. For advanced features, JavaScript libraries, and detailed examples, see REFERENCE.md. If you need to fill out a PDF form, read FORMS.md and follow its instructions.

## Default Output Format for Complex Tasks

When a user's task involves multiple tool calls (web search, data analysis, tool invocations, etc.) and produces substantial research results, but the user has **NOT specified an output format**, you should **default to generating a PDF report** using the template below. This applies to:

- Research tasks that gather information from multiple sources
- Analysis tasks that produce structured findings
- Any multi-step task where the final deliverable is a comprehensive answer

**Do NOT** default to PDF for simple Q&A, quick lookups, or tasks where the user clearly expects a chat response. Use your judgment: if the task took 5+ tool calls and produced rich, structured content, a PDF report is the appropriate default.

When defaulting to PDF output, follow the "Generate PDF Reports" workflow below — use `generate_report.py` with `report_data.json`, NOT the markdown-to-PDF approach.

---

## Quick Start — Choosing the Right Tool for Text Extraction

**Not all extractors are equal.** Pick the right one based on your PDF type:

| PDF Type | Best Tool | Why |
|---|---|---|
| Academic papers (two-column, conference/journal) | `pdftotext -layout` (poppler) | Handles column detection and character spacing reconstruction |
| Simple single-column documents | pdfplumber or pypdf | Good enough, easier to script |
| Scanned PDFs (image-based) | pytesseract + pdf2image | Needs OCR |
| Tables / structured data | pdfplumber | Best table extraction |

### Academic Papers — Use pdftotext First

Most academic PDFs (arXiv, IEEE, ACM, etc.) use two-column layouts and custom font encodings where spaces are implicit (encoded as character spacing, not space characters). Python libraries like `pypdf` and basic `pdfplumber` often produce **merged words** (e.g. `"TheConferenceonAI"` instead of `"The Conference on AI"`).

**Always prefer `pdftotext` from poppler-utils for academic papers:**

```bash
# Best option — preserves layout and column structure
pdftotext -layout input.pdf output.txt

# Alternative — raw text without layout (still handles spacing correctly)
pdftotext input.pdf output.txt
```

If `pdftotext` is not available, use PyMuPDF with sort mode:

```python
import fitz  # PyMuPDF

doc = fitz.open("document.pdf")
for page in doc:
    # sort=True reorders text by reading position (top-to-bottom, left-to-right)
    text = page.get_text("text", sort=True)
    print(text)
```

### Simple Documents — pypdf Quick Start

```python
from pypdf import PdfReader, PdfWriter

reader = PdfReader("document.pdf")
print(f"Pages: {len(reader.pages)}")

text = ""
for page in reader.pages:
    text += page.extract_text()
```

> **Warning**: pypdf's text extraction is basic. If the output has merged words or garbled column text, switch to `pdftotext -layout`.

## Generate PDF Reports (Data-Driven Template)

**MANDATORY: When creating ANY structured report (research, analysis, summary, etc.), you MUST use the pre-built professional template.** Do NOT write your own PDF generation code using reportlab, fpdf2, or any other library from scratch. The template already handles all styling, CJK fonts, layout, and pagination correctly.

This template produces business-professional quality PDFs with:
- Cover page with metadata table and disclaimer
- **Auto-generated Table of Contents with page numbers**
- Numbered section headings, dense detailed content
- Tables with auto column widths and smart alignment
- References section with numbered citations

### Step 1: Copy the generator to your workspace

**CRITICAL**: You MUST use shell `cp` to copy the script exactly as shown below. Do NOT:
- Write your own PDF generator from scratch
- Use `read_file` + `write_file` (risks stale cached version)
- Modify the generator script in any way

```bash
cp /builtin-skills/pdf/scripts/generate_report.py ./generate_report.py
```

### Step 2: Build `report_data.json`

**Two phases: write section text files, then assemble into JSON.**

**Phase 1 — Write each section as a plain text file** using `write_file`:

For each major section, `read_file` the relevant research_data, then `write_file` the section content directly:
```
read_file("research_data/literature.md")           # refresh data in context
write_file("sections/sec_01_intro.txt", "...")      # write section content
write_file("sections/sec_02_mutations.txt", "...")   # next section
...
```
Each section file should be 1,000-2,000+ words with specific data, citations, and analysis.

**NEVER write a Python script that contains section text as string literals.** The section content goes directly into .txt files via `write_file`, not into Python code. Do NOT write scripts named "generate_sections", "create_content", "build_report" etc. that embed text in Python strings. If a sandbox script fails twice, switch to direct `write_file` calls.

**Writing style — academic research report (CRITICAL):**
- Write continuous flowing prose. Each paragraph: 8--10 sentences following the pattern: topic sentence → supporting evidence with specific data → analysis/comparison → transition to next point.
- Use in-text citations [1], [2] when referencing data. These render as superscript links in the PDF. Do NOT add a "References" list at the end of each chapter — all references go in ONE final `references` section.
- Synthesize across sources: "Study A [1] reported X, while Study B [2] found Y, suggesting that Z."
- Use academic connectives: "Furthermore", "In contrast", "These findings indicate", "Notably", "Taken together".
- NEVER use numbered-point structure (e.g. "1. Topic Title\n\nParagraph. 2. Topic Title\n\nParagraph."). Instead, use `##` subheadings for structure and