Representation of syntax in LLMs through the lens of linear distance and similarity-aware entropy
Mirrored from arXiv — NLP / Computation & Language for archival readability. Support the source by reading on the original site.
Computer Science > Computation and Language
Title:Representation of syntax in LLMs through the lens of linear distance and similarity-aware entropy
Abstract:Structural probes were introduced by Hewitt and Manning to reconstruct syntactic trees from a neural language model's latent representations. They are evaluated by calculating the proportion of syntactic tree edges correctly reconstructed over an annotated corpus (as measured by undirected unlabeled attachment score). Here, we disaggregate this measure, considering undirected attachment score by label (UASL), which assesses the reconstruction accuracy of each syntactic relation separately, establishing important differences among relations that overlap linguistic distinctions. Moreover, we identify two factors that predict most of UASL's variability across relations: (i) the mean and dispersion of the linear distance (on a log scale) between the related words, and (ii) the diversity (similarity-aware entropy) of the syntactic relation's head. These results, which hold across a range of model sizes and architectures, shed light on the degree of abstraction of the representation of syntax in language models and the dependence of such representation on geometric properties of the embedding space.
| Comments: | 29 pages (9 main text, 18 appendix), 20 figures, 7 tables. Code and data: this https URL |
| Subjects: | Computation and Language (cs.CL) |
| MSC classes: | 68T50 (Primary) 68T07, 94A17 (Secondary) |
| ACM classes: | I.2.7; I.2.6; G.3 |
| Cite as: | arXiv:2608.27813 [cs.CL] |
| (or arXiv:2608.27813v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2608.27813
arXiv-issued DOI via DataCite (pending registration)
|
Submission history
From: Juan Pablo Vigneaux [view email][v1] Fri, 28 Aug 2026 01:17:18 UTC (10,008 KB)
Access Paper:
- View PDF
- HTML (experimental)
- TeX Source
References & Citations
Bookmark
Bibliographic and Citation Tools
Code, Data and Media Associated with this Article
Demos
Recommenders and Search Tools
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
More from arXiv — NLP / Computation & Language
-
Evaluating Large Language Models for Forced Outage Risk Prediction: Benefits and Comparison to Machine Learning
Sep 7
-
SharedSAE: One Feature Dictionary Across Language Models
Sep 7
-
Conformity Breaks Conformal Prediction
Sep 7
-
When Load-Balancing Goes Too Far: Expert Pruning in Over-Dispersed Mixture-of-Experts Models
Sep 7
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.