Disentangling Language Modeling and Boundaries
Mirrored from arXiv — NLP / Computation & Language for archival readability. Support the source by reading on the original site.
Computer Science > Computation and Language
Title:Disentangling Language Modeling and Boundaries
Abstract:Byte-level language models are usually argued for on the grounds of robustness, multilingual fairness, and character-level skills. We point to a different, structural advantage: because they read and write bytes, any two of them share an output space, so knowledge transfer between them is exact and independent of how either was originally tokenized. We hypothesize that the two distributions a byte-level model produces, one over the next byte, one over where its patch boundaries fall, can be disentangled and changed almost independently. A model could absorb a teacher's capability while keeping its own boundaries, or change how it places those boundaries while keeping its capabilities. We lay out the two experiments that would settle the hypothesis, alongside preliminary measurements of the properties they rest on. We argue that the community should move toward a byte-level interface as a shared standard: if the hypothesis holds, then once byte-level models are the norm, transferring capabilities and reshaping boundaries between them become cheap and routine, free of the per-model tokenizer that blocks them today.
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2608.03599 [cs.CL] |
| (or arXiv:2608.03599v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2608.03599
arXiv-issued DOI via DataCite (pending registration)
|
Access Paper:
- View PDF
- HTML (experimental)
- TeX Source
References & Citations
Bibliographic and Citation Tools
Code, Data and Media Associated with this Article
Demos
Recommenders and Search Tools
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
More from arXiv — NLP / Computation & Language
-
QEvict: Recoverable Quantized KV Eviction for Attention-Drift-Robust Long-Context Decoding
Aug 7
-
EvoHarness-RL: Learning Self-Evolving Runtime Harness for Long-Horizon LLM Agents
Aug 7
-
Reasoning Errors Have a Region and a Direction in the Residual-Stream Trajectory of LLMs
Aug 7
-
GROM: Gradient-Free Rapid One-Shot Machine Unlearning
Aug 7
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.