huggingface-llm-trainer

# TRL Training on Hugging Face Jobs

## Overview

Train language models using TRL (Transformer Reinforcement Learning) on fully managed Hugging Face infrastructure. No local GPU setup required — models train on cloud GPUs and results are automatically saved to the Hugging Face Hub.

**TRL provides multiple training methods:**
- **SFT** (Supervised Fine-Tuning) — standard instruction tuning
- **DPO** (Direct Preference Optimization) — alignment from preference data
- **GRPO** (Group Relative Policy Optimization) — online RL training
- **Reward Modeling** — train reward models for RLHF

See `references/training_methods.md` for method overviews and selection guidance.

### When to Use Unsloth

Use **Unsloth** (`references/unsloth.md`) instead of standard TRL when GPU memory is limited (~60% less VRAM), speed matters (~2x faster), training large models (>13B), or training Vision-Language Models (Unsloth has `FastVisionModel` support). See `scripts/unsloth_sft_example.py` for a production-ready training script.

## Key Directives

1. **Submit jobs via `hf jobs uv run` (CLI) or the `hf_jobs()` MCP tool if the Hugging Face MCP server is configured** — pass the training script inline, don't save to a local file unless the user explicitly requests it. If the user asks to "train a model" or "fine-tune", create the training script AND submit the job immediately.
2. **Always include Trackio** for real-time monitoring — use `scripts/` templates.
3. **Provide job details after submission**: job ID, monitoring URL, estimated time; note the user can request status checks later.
4. **Use example scripts as templates**: `scripts/train_sft_example.py`, `scripts/train_dpo_example.py`, etc.

## Local Script Execution

Repository scripts use PEP 723 inline dependencies. Run them with `uv run`:
```bash
uv run scripts/estimate_cost.py --help
uv run scripts/dataset_inspector.py --help
```

## Prerequisites Checklist

**Account & Authentication:**
- Hugging Face account with Pro/Team/Enterprise plan (Jobs require a paid plan); authenticated login.
- **HF_TOKEN for Hub push is CRITICAL** — the training environment is ephemeral, so results are lost unless pushed to the Hub. Token must have write permissions. Pass `secrets={"HF_TOKEN": "$HF_TOKEN"}` in the job config.

**Dataset Requirements:**
- Must exist on the Hub or be loadable via `datasets.load_dataset()`.
- Format must match the training method (SFT: messages/text/prompt-completion; DPO: chosen/rejected; GRPO: prompt-only). **Always validate unknown datasets first** (see Dataset Validation below).
- Size appropriate for hardware (demo: 50-100 examples on t4-small; production: 1K-10K+ on a10g-large/a100-large).

**Critical Settings:**
- Timeout must exceed expected training time — default 30min is too short; minimum recommended 1-2 hours. The job fails and loses all progress if the timeout is exceeded.
- Hub push must be enabled: `push_to_hub=True`, `hub_model_id="username/model-name"`, `secrets={"HF_TOKEN": "$HF_TOKEN"}`.

## Asynchronous Jobs

Training jobs run in the background and can take hours. After submitting: report the job ID, monitoring URL, and estimated time; wait for the user to request status checks rather than polling. Initial logs can take 30-60 seconds to appear.

## Quick Start

**Sequence length:** TRL config classes use `max_length` (not `max_seq_length`). Default is `max_length=1024` (truncates from right) — override higher for longer context, lower under memory constraints, or `None` for vision models (to avoid cutting image tokens).

### Approach 1: UV Scripts (default choice)

UV scripts use PEP 723 inline dependencies for clean, self-contained training:

```python
hf_jobs("uv", {
    "script": """
# /// script
# dependencies = ["trl>=0.12.0", "peft>=0.7.0", "trackio"]
# ///
from datasets import load_dataset
from peft import LoraConfig
from trl import SFTTrainer, SFTConfig
import trackio

dataset = load_dataset("trl-lib/Capybara", split="train")
dataset_split = dataset.train_test_split(test_size=0.1, seed=42)

trainer = SFTTrainer(
    model="Qwen/Qwen2.5-0.5B",
    train_dataset=dataset_split["train"],
    eval_dataset=dataset_split["test"],
    peft_config=LoraConfig(r=16, lora_alpha=32),
    args=SFTConfig(
        output_dir="my-model", push_to_hub=True, hub_model_id="username/my-model",
        num_train_epochs=3, eval_strategy="steps", eval_steps=50,
        report_to="trackio", project="my_project", run_name="my_run",
    ),
)
trainer.train()
trainer.push_to_hub()
""",
    "flavor": "a10g-large",
    "timeout": "2h",
    "secrets": {"HF_TOKEN": "$HF_TOKEN"},
})
```

The `script` parameter accepts inline code or a publicly-accessible/Hub/GitHub/Gist URL — **local file paths do not work** (jobs run in isolated containers with no access to the local filesystem). To use a local script, upload it to the Hub first (`hf upload ...`) and reference its resolved URL.

### Approach 2: TRL Maintained Scripts

Run TRL's battle-tested example scripts directly from a URL, passing CLI-style `script_args` (`--model_name_or_path`, `--dataset_name`, `--output_dir`, `--push_to_hub`, `--hub_model_id`). Available at https://github.com/huggingface/trl/tree/main/examples/scripts.

### Approach 3: HF Jobs CLI

When no `hf_jobs`-style tool is available, use the `hf jobs` CLI directly. **Flags must come before the script URL**, the subcommand order is `hf jobs uv run` (not `run uv`), and use `--secrets` (plural):

```bash
hf jobs uv run \
  --flavor a10g-large --timeout 2h --secrets HF_TOKEN \
  "https://huggingface.co/user/repo/resolve/main/train.py"
```

Check status: `hf jobs ps`, `hf jobs logs <job-id>`, `hf jobs inspect <job-id>`, `hf jobs cancel <job-id>`.

### Approach 4: TRL Jobs Package

`uvx trl-jobs sft --model_name Qwen/Qwen2.5-0.5B --dataset_name trl-lib/Capybara` gives pre-configured defaults, automatic Trackio integration, and automatic Hub push — best for terminal-only, quick local experimentation. Repository: https://github.com/huggingf