Tokenizer Expansion: Upgrading a Model's Tokenizer in Place - LFM2.5-8B-A1B

Today, we're sharing the recipe behind the new tokenizer in LFM2.5-8B-A1B. It upgrades a pre-trained model's tokenizer in place, without retraining from scratch. We doubled the vocabulary from 65K to 128K to fix the languages our original tokenizer split too finely.

Blog: liquid.ai/blog/tokenizer-expansion
Technical report: arxiv.org/abs/2607.15232
Hugging Face: huggingface.co/LiquidAI/LFM2.5-8B-A1B