The Loss Does Not See the Basis, But Adam Does [R]

In a factored model W = UV^T, the loss is invariant to rotations (U,V) → (UQ, VQ). GD respects that. Adam's per-coordinate second moment doesn't, because it depends on which basis you happen to write the factors in.

The claim is that this one property is what sorts optimizers into keeping or losing GD's implicit low-rank bias.

I ran nine update rules on underdetermined matrix sensing, all compared at matched training loss so nothing wins by fitting less. Two clean clusters. GD, shared-scalar Adam, Muon and Shampoo keep the bias. Adam, RMSProp, Lion, signum and Adafactor lose it.

To find the actual lever, there's a one-parameter family that turns Adam's denominator from per-coordinate into a single shared scalar. Recovery improves monotonically along it, which pins the damage on the anisotropy rather than on adaptivity in general.

Muon was the part I didn't expect. It's exact on truly low-rank targets, then degrades fastest as you add a spectral tail and cedes to GD in a crossover near 4% tail energy. Recent work disagrees about Muon here, with some reporting a strong spectral simplicity bias and others reporting it fits spurious features in deep-linear models. My sweep shows both, on the same axis.

I also ran the criterion on my own earlier optimizer and found its per-coordinate clip was breaking the structure it existed to inject. Global norm clip instead: recovery error 0.347 → 0.220.

One caveat up front. The 43-44% held-out error reduction on hyperspectral data uses a train-only learning rate rule, and that rule hands Adam the worst rate on its own grid. Let each method pick its own best rate and the gap is considerably smaller (Appendix D.6). I kept the train-only rule since selecting on held-out data is the exact bias the experiment exists to avoid, but the mechanism is the claim, not the number.

Theory covers memoryless rules only. Momentum is empirical here, not proved.

Paper:https://arxiv.org/abs/2608.05136

Code, logs, seeds:https://github.com/idevender/loss-basis-adam

Happy to take the "you should have just tuned Adam harder" objections!