Subword tokenization is an essential part of modern large language models (LLMs), yet its specific contributions to training efficiency and model performance remain poorly understood. In this work, we decouple the effects of subword tokenization by isolating them within a controlled byte-level pretraining pipeline. We formulate and test hypotheses across various dimensions, including sample throughput, vocabulary scaling, and the linguistic prior of subword boundaries. By simulating these effects in a byte-level setting, we refine our understanding of why subword models outperform raw byte models and offer insights to improve the pretraining of future byte-level and subword models. Specifically, our experiments highlight the critical role of increased training throughput and the integration of subword boundaries as either explicit priors or inductive biases.</p>\n","updatedAt":"2026-05-21T20:35:27.268Z","author":{"_id":"60d35154d7b174177faabd55","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/60d35154d7b174177faabd55/_if2cJtR1Um5R5xavy6Kk.jpeg","fullname":"théo gigant","name":"gigant","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":48,"isUserFollowing":false,"primaryOrg":{"avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/6317aade83d8d2fd903192d9/tPLjYEeP6q1w0j_G2TJG_.png","fullname":"NousResearch","name":"NousResearch","type":"org","isHf":false,"details":"We are dedicated to advancing the field of natural language processing, in collaboration with the open-source community, through bleeding-edge research and a commitment to symbiotic development.","plan":"team"}}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.9013359546661377},"editors":["gigant"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/60d35154d7b174177faabd55/_if2cJtR1Um5R5xavy6Kk.jpeg"],"reactions":[],"isReport":false}},{"id":"6a0fb554a339b290b06b73f4","author":{"_id":"63d3e0e8ff1384ce6c5dd17d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg","fullname":"Librarian Bot (Bot)","name":"librarian-bot","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":358,"isUserFollowing":false},"createdAt":"2026-05-22T01:45:56.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"This is an automated message from the [Librarian Bot](https://huggingface.co/librarian-bots). I found the following papers similar to this paper. \n\nThe following papers were recommended by the Semantic Scholar API \n\n* [Efficient Pre-Training with Token Superposition](https://huggingface.co/papers/2605.06546) (2026)\n* [The Efficiency Gap in Byte Modeling](https://huggingface.co/papers/2605.12928) (2026)\n* [Cross-Tokenizer LLM Distillation through a Byte-Level Interface](https://huggingface.co/papers/2604.07466) (2026)\n* [KazByte: Adapting Qwen models to Kazakh via Byte-level Adapter](https://huggingface.co/papers/2603.27859) (2026)\n* [Fast Byte Latent Transformer](https://huggingface.co/papers/2605.08044) (2026)\n* [Think in Sentences: Explicit Sentence Boundaries Enhance Language Model's Capabilities](https://huggingface.co/papers/2604.10135) (2026)\n* [MIXAR: Scaling Autoregressive Pixel-based Language Models to Multiple Languages and Scripts](https://huggingface.co/papers/2604.11575) (2026)\n\n\n Please give a thumbs up to this comment if you found it helpful!\n\n If you want recommendations for any Paper on Hugging Face checkout [this](https://huggingface.co/spaces/librarian-bots/recommend_similar_papers) Space\n\n You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: `@librarian-bot recommend`","html":"<p>This is an automated message from the <a href=\"https://huggingface.co/librarian-bots\">Librarian Bot</a>. I found the following papers similar to this paper. </p>\n<p>The following papers were recommended by the Semantic Scholar API </p>\n<ul>\n<li><a href=\"https://huggingface.co/papers/2605.06546\">Efficient Pre-Training with Token Superposition</a> (2026)</li>\n<li><a href=\"https://huggingface.co/papers/2605.12928\">The Efficiency Gap in Byte Modeling</a> (2026)</li>\n<li><a href=\"https://huggingface.co/papers/2604.07466\">Cross-Tokenizer LLM Distillation through a Byte-Level Interface</a> (2026)</li>\n<li><a href=\"https://huggingface.co/papers/2603.27859\">KazByte: Adapting Qwen models to Kazakh via Byte-level Adapter</a> (2026)</li>\n<li><a href=\"https://huggingface.co/papers/2605.08044\">Fast Byte Latent Transformer</a> (2026)</li>\n<li><a href=\"https://huggingface.co/papers/2604.10135\">Think in Sentences: Explicit Sentence Boundaries Enhance Language Model's Capabilities</a> (2026)</li>\n<li><a href=\"https://huggingface.co/papers/2604.11575\">MIXAR: Scaling Autoregressive Pixel-based Language Models to Multiple Languages and Scripts</a> (2026)</li>\n</ul>\n<p> Please give a thumbs up to this comment if you found it helpful!</p>\n<p> If you want recommendations for any Paper on Hugging Face checkout <a href=\"https://huggingface.co/spaces/librarian-bots/recommend_similar_papers\">this</a> Space</p>\n<p> You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: <code><span class=\"SVELTE_PARTIAL_HYDRATER contents\" data-target=\"UserMention\" data-props=\"{&quot;user&quot;:&quot;librarian-bot&quot;}\"><span class=\"inline-block\"><span class=\"contents\"><a href=\"/librarian-bot\">@<span class=\"underline\">librarian-bot</span></a></span> </span></span> recommend</code></p>\n","updatedAt":"2026-05-22T01:45:56.967Z","author":{"_id":"63d3e0e8ff1384ce6c5dd17d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg","fullname":"Librarian Bot (Bot)","name":"librarian-bot","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":358,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.7156301736831665},"editors":["librarian-bot"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2604.27263","authors":[{"_id":"6a0f6c2da53a61ce2e422b4c","name":"Théo Gigant","hidden":false},{"_id":"6a0f6c2da53a61ce2e422b4d","name":"Bowen Peng","hidden":false},{"_id":"6a0f6c2da53a61ce2e422b4e","name":"Jeffrey Quesnelle","hidden":false}],"publishedAt":"2026-05-14T00:00:00.000Z","submittedOnDailyAt":"2026-05-21T00:00:00.000Z","title":"Decoupling the Benefits of Subword Tokenization for Language Model Training via Byte-level Simulation","submittedOnDailyBy":{"_id":"60d35154d7b174177faabd55","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/60d35154d7b174177faabd55/_if2cJtR1Um5R5xavy6Kk.jpeg","isPro":false,"fullname":"théo gigant","user":"gigant","type":"user","name":"gigant"},"summary":"Subword tokenization is an essential part of modern large language models (LLMs), yet its specific contributions to training efficiency and model performance remain poorly understood. In this work, we decouple the effects of subword tokenization by isolating them within a controlled byte-level pretraining pipeline. We formulate and test hypotheses across various dimensions, including sample throughput, vocabulary scaling, and the linguistic prior of subword boundaries. By simulating these effects in a byte-level setting, we refine our understanding of why subword models outperform raw byte models and offer insights to improve the pretraining of future byte-level and subword models. Specifically, our experiments highlight the critical role of increased training throughput and the integration of subword boundaries as either explicit priors or inductive biases.","upvotes":1,"discussionId":"6a0f6c2da53a61ce2e422b4f","ai_summary":"Research investigates subword tokenization's impact on LLM training efficiency and performance through controlled byte-level pretraining experiments, revealing key factors in training throughput and linguistic priors.","ai_keywords":["subword tokenization","large language models","byte-level pretraining","sample throughput","vocabulary scaling","linguistic prior","inductive biases"],"organization":{"_id":"643b858ba856622f9790cc66","name":"NousResearch","fullname":"NousResearch","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/6317aade83d8d2fd903192d9/tPLjYEeP6q1w0j_G2TJG_.png"}},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"60d35154d7b174177faabd55","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/60d35154d7b174177faabd55/_if2cJtR1Um5R5xavy6Kk.jpeg","isPro":false,"fullname":"théo gigant","user":"gigant","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"organization":{"_id":"643b858ba856622f9790cc66","name":"NousResearch","fullname":"NousResearch","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/6317aade83d8d2fd903192d9/tPLjYEeP6q1w0j_G2TJG_.png"},"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2604/2604.27263.md"}">

Get this paper in your agent:

Don't have the latest CLI?

Cite arxiv.org/abs/2604.27263 in a model README.md to link it from this page.

Cite arxiv.org/abs/2604.27263 in a dataset README.md to link it from this page.

Cite arxiv.org/abs/2604.27263 in a Space README.md to link it from this page.

Add this paper to a collection to link it from this page.