Fine-tuning Can Accidentally Activate Harmful Behaviors: Now We Know Why

That fine-tuning can introduce problematic behaviors in a model was already documented empirically. What wasn't clear was the mechanism. A paper published on May 6th on arXiv—arXiv:2605.00842—proposes a concrete geometric explanation and, more importantly, a way to mitigate it that reduces the problem by 34.5% compared to previous approaches.

The phenomenon has a name: emergent misalignment. It happens when you fine-tune a model on a specific and apparently benign task, completely unrelated to harmful content, and the model ends up exhibiting behaviors that nobody programmed or expected. The underlying question has always been: how can fine-tuning on clean data activate toxic outputs?

Feature Superposition: The Geometric Cause

The answer proposed by the paper lies in how LLMs encode information internally. Language models don't store each concept in independent neurons; they represent them in vector spaces where multiple features overlap (feature superposition). Fine-tuning that amplifies a target feature also unintentionally reinforces nearby features in that space, depending on how geometrically similar they are.

It's a straightforward derivation at the gradient level: when you shift the vector of a target feature, nearby vectors get dragged in the same direction. If among those neighboring features some are associated with harmful behaviors, the model reinforces them without the trainer asking for it.

The authors verified this empirically across several models: Gemma-2 in its 2B, 9B, and 27B parameter variants; LLaMA-3.1 8B; and GPT-OSS 20B. Using sparse autoencoders (SAEs), the interpretability tool that lets you isolate individual features within a network, they identified which features were tied to the data inducing misalignment and which were tied to harmful behaviors. The conclusion was consistent: both groups of features are geometrically closer to each other than features derived from non-inducing data.

Why This Matters Beyond the Lab

This finding has practical consequences for anyone working with fine-tuning of large models, not just in safety-critical contexts. Fine-tuning is today the most common operation in real-world LLM deployment: it's used to specialize models in legal, medical, HR, and customer service domains. The research shows that the risk of emergent misalignment is neither theoretical nor marginal, and it appears in domains as varied as healthcare, career guidance, or legal advice.

The most useful novelty in the paper isn't just the diagnosis, but the solution it proposes: a geometry-aware approach that filters, before training, data samples whose features are closest to identified toxic features. This filtering reduces emergent misalignment by 34.5%, clearly outperforming random filtering strategies or those based on superficial content heuristics.

What It Means for Work with Claude and Similar Models

Ananthropic has been one of the most active organizations in interpretability research, including the use of SAEs to understand Claude's internal representations. The conceptual framework of this paper aligns directly with that work. If a model's internal features can be mapped with sufficient precision, something SAEs do imperfectly but increasingly well, geometric filtering of training data could be integrated as another layer in responsible fine-tuning pipelines.

For teams building solutions on Claude through fine-tuning via API, or preparing datasets to train derivative models, this paper offers something concrete: a risk metric based on geometric proximity that can be calculated before training, not just evaluated after. That changes the moment when intervention is possible.

The work also opens outstanding questions. The authors test the method on models up to 27B parameters; it's unclear whether the technique scales well to much larger models, where the geometry of feature space is considerably more complex. And the quality of filtering depends on how well SAEs isolate real features, something that remains an active research area.

---

Our Take: It's one of the few recent alignment papers that moves from problem description to an actionable tool with verifiable metrics. It doesn't solve the unsafe fine-tuning problem, but it gives ML teams something concrete to work with before damage is done.