simpo-training
SimPO is a reference-free preference optimization method for language model alignment that delivers superior performance compared to DPO without requiring a separate reference model. Use it when training LLMs to follow human preferences more efficiently, as it requires fewer computational resources and demonstrates measurable improvements on benchmarks like AlpacaEval while simplifying the training pipeline compared to DPO or PPO approaches.
git clone --depth 1 https://github.com/Orchestra-Research/AI-Research-SKILLs /tmp/simpo-training && cp -r /tmp/simpo-training/06-post-training/simpo ~/.claude/skills/simpo-trainingSKILL.md
# SimPO - Simple Preference Optimization ## Quick start SimPO is a reference-free preference optimization method that outperforms DPO without needing a reference model. **Installation**: ```bash # Create environment conda create -n simpo python=3.10 && conda activate simpo # Install PyTorch 2.2.2 # Visit: https://pytorch.org/get-started/locally/ # Install alignment-handbook git clone https://github.com/huggingface/alignment-handbook.git cd alignment-handbook python -m pip install . # Install Flash Attention 2 python -m pip install flash-attn --no-build-isolation ``` **Training** (Mistral 7B): ```bash ACCELERATE_LOG_LEVEL=info accelerate launch \ --config_file accelerate_configs/deepspeed_zero3.yaml \ scripts/run_simpo.py \ training_configs/mistral-7b-base-simpo.yaml ``` ## Common workflows ### Workflow 1: Train from base model (Mistral 7B) **Config** (`mistral-7b-base-simpo.yaml`): ```yaml # Model model_name_or_path: mistralai/Mistral-7B-v0.1 torch_dtype: bfloat16 # Dataset dataset_mixer: HuggingFaceH4/ultrafeedback_binarized: 1.0 dataset_splits: - train_prefs - test_prefs # SimPO hyperparameters beta: 2.0 # Reward scaling (2.0-10.0) gamma_beta_ratio: 0.5 # Target margin (0-1) loss_type: sigmoid # sigmoid or hinge sft_weight: 0.0 # Optional SFT regularization # Training learning_rate: 5e-7 # Critical: 3e-7 to 1e-6 num_train_epochs: 1 per_device_train_batch_size: 1 gradient_accumulation_steps: 8 # Output output_dir: ./outputs/mistral-7b-simpo ``` **Launch training**: ```bash accelerate launch --config_file accelerate_configs/deepspeed_zero3.yaml \ scripts/run_simpo.py training_configs/mistral-7b-base-simpo.yaml ``` ### Workflow 2: Fine-tune instruct model (Llama 3 8B) **Config** (`llama3-8b-instruct-simpo.yaml`): ```yaml model_name_or_path: meta-llama/Meta-Llama-3-8B-Instruct dataset_mixer: argilla/ultrafeedback-binarized-preferences-cleaned: 1.0 beta: 2.5 gamma_beta_ratio: 0.5 learning_rate: 5e-7 sft_weight: 0.1 # Add SFT loss to preserve capabilities num_train_epochs: 1 per_device_train_batch_size: 2 gradient_accumulation_steps: 4 output_dir: ./outputs/llama3-8b-simpo ``` **Launch**: ```bash accelerate launch --config_file accelerate_configs/deepspeed_zero3.yaml \ scripts/run_simpo.py training_configs/llama3-8b-instruct-simpo.yaml ``` ### Workflow 3: Reasoning-intensive tasks (lower LR) **For math/code tasks**: ```yaml model_name_or_path: deepseek-ai/deepseek-math-7b-base dataset_mixer: argilla/distilabel-math-preference-dpo: 1.0 beta: 5.0 # Higher for stronger signal gamma_beta_ratio: 0.7 # Larger margin learning_rate: 3e-7 # Lower LR for reasoning sft_weight: 0.0 num_train_epochs: 1 per_device_train_batch_size: 1 gradient_accumulation_steps: 16 ``` ## When to use vs alternatives **Use SimPO when**: - Want simpler training than DPO (no reference model) - Have preference data (chosen/rejected pairs) - Need better performance than DPO - Limited compute resources - Single-node training sufficient **Algorithm selection**: - **SimPO**: Simplest, best performance, no reference model - **DPO**: Need reference model baseline, more conservative - **PPO**: Maximum control, need reward model, complex setup - **GRPO**: Memory-efficient RL, no critic **Use alternatives instead**: - **OpenRLHF**: Multi-node distributed training, PPO/GRPO - **TRL**: Need multiple methods in one framework - **DPO**: Established baseline comparison ## Common issues **Issue: Loss divergence** Reduce learning rate: ```yaml learning_rate: 3e-7 # Reduce from 5e-7 ``` Reduce beta: ```yaml beta: 1.0 # Reduce from 2.0 ``` **Issue: Model forgets capabilities** Add SFT regularization: ```yaml sft_weight: 0.1 # Add SFT loss component ``` **Issue: Poor preference separation** Increase beta and margin: ```yaml beta: 5.0 # Increase from 2.0 gamma_beta_ratio: 0.8 # Increase from 0.5 ``` **Issue: OOM during training** Reduce batch size: ```yaml per_device_train_batch_size: 1 gradient_accumulation_steps: 16 # Maintain effective batch ``` Enable gradient checkpointing: ```yaml gradient_checkpointing: true ``` ## Advanced topics **Loss functions**: See [references/loss-functions.md](references/loss-functions.md) for sigmoid vs hinge loss, mathematical formulations, and when to use each. **Hyperparameter tuning**: See [references/hyperparameters.md](references/hyperparameters.md) for beta, gamma, learning rate selection guide, and model-size-specific recommendations. **Dataset preparation**: See [references/datasets.md](references/datasets.md) for preference data formats, quality filtering, and custom dataset creation. ## Hardware requirements - **GPU**: NVIDIA A100/H100 recommended - **VRAM**: - 7B model: 1× A100 40GB (DeepSpeed ZeRO-3) - 8B model: 2× A100 40GB - 70B model: 8× A100 80GB - **Single-node**: DeepSpeed ZeRO-3 sufficient - **Mixed precision**: BF16 recommended **Memory optimization**: - DeepSpeed ZeRO-3 (default config) - Gradient checkpointing - Flash Attention 2 ## Resources - Paper: https://arxiv.org/abs/2405.14734 (NeurIPS 2024) - GitHub: https://github.com/princeton-nlp/SimPO - Models: https://huggingface.co/princeton-nlp - Alignment Handbook: https://github.com/huggingface/alignment-handbook
Orchestrates end-to-end autonomous AI research projects using a two-loop architecture. The inner loop runs rapid experiment iterations with clear optimization targets. The outer loop synthesizes results, identifies patterns, and steers research direction. Routes to domain-specific skills for execution, supports continuous agent operation via Claude Code /loop and OpenClaw heartbeat, and produces research presentations and papers. Use when starting a research project, running autonomous experiments, or managing a multi-hypothesis research effort.
Implements and trains LLMs using Lightning AI's LitGPT with 20+ pretrained architectures (Llama, Gemma, Phi, Qwen, Mistral). Use when need clean model implementations, educational understanding of architectures, or production fine-tuning with LoRA/QLoRA. Single-file implementations, no abstraction layers.
State-space model with O(n) complexity vs Transformers' O(n²). 5× faster inference, million-token sequences, no KV cache. Selective SSM with hardware-aware design. Mamba-1 (d_state=16) and Mamba-2 (d_state=128, multi-head). Models 130M-2.8B on HuggingFace.
Educational GPT implementation in ~300 lines. Reproduces GPT-2 (124M) on OpenWebText. Clean, hackable code for learning transformers. By Andrej Karpathy. Perfect for understanding GPT architecture from scratch. Train on Shakespeare (CPU) or OpenWebText (multi-GPU).
RNN+Transformer hybrid with O(n) inference. Linear time, infinite context, no KV cache. Train like GPT (parallel), infer like RNN (sequential). Linux Foundation AI project. Production at Windows, Office, NeMo. RWKV-7 (March 2025). Models up to 14B parameters.
Provides PyTorch-native distributed LLM pretraining using torchtitan with 4D parallelism (FSDP2, TP, PP, CP). Use when pretraining Llama 3.1, DeepSeek V3, or custom models at scale from 8 to 512+ GPUs with Float8, torch.compile, and distributed checkpointing.
Fast tokenizers optimized for research and production. Rust-based implementation tokenizes 1GB in <20 seconds. Supports BPE, WordPiece, and Unigram algorithms. Train custom vocabularies, track alignments, handle padding/truncation. Integrates seamlessly with transformers. Use when you need high-performance tokenization or custom tokenizer training.
Language-independent tokenizer treating text as raw Unicode. Supports BPE and Unigram algorithms. Fast (50k sentences/sec), lightweight (6MB memory), deterministic vocabulary. Used by T5, ALBERT, XLNet, mBART. Train on raw text without pre-tokenization. Use when you need multilingual support, CJK languages, or reproducible tokenization.