Where Quality Breaks in Compressed Short-Text Generation: Staged Bottleneck Localization
Mirrored from arXiv — NLP / Computation & Language for archival readability. Support the source by reading on the original site.
Computer Science > Computation and Language
arXiv:2607.24176 (cs)
[Submitted on 27 Jul 2026]
Title:Where Quality Breaks in Compressed Short-Text Generation: Staged Bottleneck Localization
View a PDF of the paper titled Where Quality Breaks in Compressed Short-Text Generation: Staged Bottleneck Localization, by Alexey Gavrilov and 2 other authors
View PDF
HTML (experimental)
Abstract:Compressed short-text generators can fail in two different places: the codec may discard information before generation starts, or the latent generator may produce weak codes. Without separating these failure modes, researchers can spend compute improving the wrong component. We study this problem in a controlled 64-to-16 TinyStories case study built from a hierarchical VQ-VAE-2 codec and a masked discrete diffusion generator (MDLM). We use a staged validation protocol that separates codec reconstruction fidelity, latent generation quality, and auxiliary latent diagnostics under one shared external GPT-2 scorer, while reporting complementary semantic metrics for the geometry study. In the tested configuration, codec reconstruction alone raises median external perplexity from 15.17 to 27.36 (+80.4%) and p95 from 25.10 to 98.91 (+294.1%), showing that the dominant quality loss appears before latent generation begins. Under the same scorer, code-space MDLM remains materially stronger than token-space diffusion, reducing mean, median, and p95 by 32.9%, 30.9%, and 36.6%, respectively. Geometry-aware regularization improves local latent proxies but does not improve decoded-text metrics in the available runs. The contribution is methodological rather than algorithmic: the paper presents a reusable staged diagnosis for one concrete pipeline and shows that, in this setting, codec fidelity rather than latent denoising sets the practical quality ceiling.
| Comments: | 8 pages, 3 figures, 14 tables. Published in the Proceedings of FRUCT'39 |
| Subjects: | Computation and Language (cs.CL); Machine Learning (cs.LG) |
| Cite as: | arXiv:2607.24176 [cs.CL] |
| (or arXiv:2607.24176v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2607.24176
arXiv-issued DOI via DataCite (pending registration)
|
|
| Journal reference: | Proceedings of the 39th Conference of the Open Innovations Association FRUCT (FRUCT'39), 2026, pp. 69-76 |
| Related DOI: | https://doi.org/10.23919/FRUCT70069.2026.11506553
DOI(s) linking to related resources
|
Full-text links:
Access Paper:
- View PDF
- HTML (experimental)
- TeX Source
View a PDF of the paper titled Where Quality Breaks in Compressed Short-Text Generation: Staged Bottleneck Localization, by Alexey Gavrilov and 2 other authors
References & Citations
Loading...
Bookmark
Bibliographic Tools
Code, Data, Media
Demos
Related Papers
About arXivLabs
Bibliographic and Citation Tools
Bibliographic Explorer Toggle
Bibliographic Explorer (What is the Explorer?)
Connected Papers Toggle
Connected Papers (What is Connected Papers?)
Litmaps Toggle
Litmaps (What is Litmaps?)
scite.ai Toggle
scite Smart Citations (What are Smart Citations?)
Code, Data and Media Associated with this Article
alphaXiv Toggle
alphaXiv (What is alphaXiv?)
Links to Code Toggle
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
DagsHub Toggle
DagsHub (What is DagsHub?)
GotitPub Toggle
Gotit.pub (What is GotitPub?)
Huggingface Toggle
Hugging Face (What is Huggingface?)
ScienceCast Toggle
ScienceCast (What is ScienceCast?)
Demos
Replicate Toggle
Replicate (What is Replicate?)
Spaces Toggle
Hugging Face Spaces (What is Spaces?)
Spaces Toggle
TXYZ.AI (What is TXYZ.AI?)
Recommenders and Search Tools
Link to Influence Flower
Influence Flower (What are Influence Flowers?)
Core recommender toggle
CORE Recommender (What is CORE?)
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
More from arXiv — NLP / Computation & Language
-
Unified Hallucination Fuzzing for Multimodal Large Language Models
Aug 11
-
DocAtlas: Long-Document Understanding as Mutable-State Interaction
Aug 11
-
WuYuEval: A Multi-Level Benchmark for Large Language Models in Solid Waste Management
Aug 11
-
Search-G1: Grounded Search Agents via Representation-Based Intrinsic Rewards
Aug 11
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.