Pruning neural networks with the Fisher information metric

Magnitude pruning, meaning delete the smallest weights and retrain, has been the default method for decades for an inelegant reason: it is trivial to implement and it works. A paper published on 16 September on arXiv (arXiv:2609.16129) argues that this criterion is not an arbitrary heuristic but the first rung of a hierarchy, and that climbing one rung improves results across every architecture and dataset combination the authors test.

The approach starts from a geometric observation. Pruning a parameter means setting it to zero, and setting a parameter to zero means displacing the model onto the hypersurface where that parameter vanishes. The question stops being which weight is smallest and becomes which displacement is shortest.

The hierarchy that falls out of the geometry

"Short" in model space is not measured in weight units, it is measured with the Fisher information metric, which weights each direction by how much the model's output distribution changes when moving along it. The minimal distance to the pruning hypersurface is then a geodesic distance under that metric, and that distance is what reflects the true change in the model and in its performance.

That is where the work gets interesting: by analysing progressively more faithful approximations of that geodesic, a natural ordering of optimality appears. The crudest approximation reproduces plain old magnitude pruning. The next one corresponds to using local Fisher information alone. Above those sit the new schemes the paper proposes. Put differently, the classic method was not a hunch: it was the degenerate case of something better.

What they test

The experimental protocol is more complete than the average in pruning literature:

1. Two architecture families: fully connected networks and vision transformers.
2. Two datasets: MNIST and CIFAR-10.
3. The complete pruning range, from 0 to 100 percent, not just the comfortable stretch.
4. Five random seeds per configuration.

The reported result is that the method outperforms magnitude pruning and pruning by local Fisher information alone in every architecture and dataset combination considered, on accuracy. The abstract published on arXiv cuts off before detailing the second metric they evaluate, so that figure has to be looked up in the PDF.

Sweeping the full pruning range matters more than it seems. Many papers in the area report only the sweet spot, that 50 to 90 percent band where almost any criterion holds up, and hide what happens at the extremes.

What the paper does not prove

MNIST and CIFAR-10 are small benchmarks. The optimality ordering holding on a vision transformer over CIFAR-10 does not guarantee it holds on a language model with billions of parameters, where structured pruning and quantisation compete for the same memory budget and where the retraining afterwards costs real money.

There is also the cost question. The full Fisher matrix is quadratic in the number of parameters, which is why in practice everyone uses the diagonal. If the hierarchy of approximations climbs in quality but also in compute cost, the useful rung will depend on each team's budget.

Who it is useful for

This does not apply tomorrow to a Claude deployment over the API: nothing gets pruned there, the weights belong to Anthropic. The real audience is teams training or fine tuning their own models that need to fit them into memory constrained devices, plus anyone researching compression who wants a theoretical frame instead of a catalogue of tricks.

We like the kind of paper that proposes not a new trick but a frame in which the previous trick is explained, because that tends to age better. That said, between MNIST and a model in production there are several orders of magnitude, and that distance has to be measured too.