Next-Latent Prediction Transformers [R]

Microsoft Research Preprint

Next-token prediction is myopic. What if transformers learn to predict their own next latent state?

Microsoft Research present Next-Latent Prediction (NextLat): a self-supervised learning method that teaches transformers to form compact world models for reasoning and planning. It also unlocks up to 3.3x faster inference via self-speculative decoding!

On top of next-token prediction, NextLat trains the transformer to predict its own next latent state given the current latent state and next token.

NextLat has a few key benefits:

1. Representation Learning: NextLat encourages transformers to compress history into compact belief states.
2. Better Data Efficiency: predicting in latent space provides denser supervision than predicting one-hot tokens.
3. Faster Inference: via recursive multi-step lookahead.

I'm super excited about this work. Please do check it out below:

💬 Blog: https://jaydenteoh.github.io/blog/2026/nextlat
💻 Code: https://github.com/JaydenTeoh
📝 Paper: https://arxiv.org/abs/2511.05963