Informational Antilocality and the Locality Bias in LLMs
Mirrored from arXiv — NLP / Computation & Language for archival readability. Support the source by reading on the original site.
Computer Science > Computation and Language
Title:Informational Antilocality and the Locality Bias in LLMs
Abstract:We consider the ability of transformer-based language models (LLMs) to learn what we call k-antilocal languages, i.e., languages that have no mutual information across any span of $k$ contiguous symbols. We construct such languages with increasing $k$, finding that LLMs trained on them achieve comparable cross-entropy loss regardless of antilocality, but converge more slowly on more antilocal languages. Our findings support the idea that non-local dependencies are more difficult to learn, but the evidence for this bias comes from learning speed rather than learning success.
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2608.27760 [cs.CL] |
| (or arXiv:2608.27760v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2608.27760
arXiv-issued DOI via DataCite (pending registration)
|
Access Paper:
- View PDF
- HTML (experimental)
- TeX Source
References & Citations
Bookmark
Bibliographic and Citation Tools
Code, Data and Media Associated with this Article
Demos
Recommenders and Search Tools
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
More from arXiv — NLP / Computation & Language
-
"As a Language Model...": Chat Template Switches LLM Self-Referential Voice and Activation Steering Reproduces It
Sep 23
-
Trains but Doesn't Learn: A Post-Training Delivery Benchmark for LLM Agents as Forward-Deployed Engineers
Sep 23
-
Rewired or Gated? How Instruction Tuning Shapes Knowledge-Conflict Circuits in LLMs
Sep 23
-
Efficient Cost-Aware LLM Evaluation via Bayesian Bandit Gittins Indices
Sep 23
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.