CAVEWOMAN: How Large Language Models Behave Under Linguistic Input and Output Compression
Mirrored from Hugging Face Daily Papers for archival readability. Support the source by reading on the original site.
CAVEWOMAN: How Large Language Models Behave Under Linguistic Input and Output Compression
Abstract
Two-channel evaluation shows output compression reduces costs while input compression increases costs and degrades accuracy across models and datasets.
"Talk short. Drop grammar. Save token." This caveman style is widely promoted as a way to cut inference cost, but whether it actually saves anything depends on which channel (the user's prompt or the model's response) is being compressed. We present Cavewoman, a two-channel evaluation protocol that scores every generation on task accuracy, realized per-item cost, and reference-text agreement against the model's unconstrained reference. We evaluate eight models on five datasets at five reduction levels, with both channels measured on the same items. Output compression cuts realized cost on most API models (1.4-2.4x per model, up to 3x in the best case) and on all four open-weight models under public-tier pricing. Input compression has the opposite effect, a strict lose-lose: it raises net cost rather than lowering it (~1.15x on the five-benchmark mean, up to 1.8x on the worst dataset and 2.7x under stronger compression), because models compensate with longer responses even as accuracy collapses. Under the same setting, surface text diverges from the unconstrained reference: on the non-reasoning models, roughly half of all generations are correct yet their surface text no longer entails the model's own unconstrained baseline generation. The divergence survives length-controlled re-scoring, multiple-comparisons correction, and replication under complementary semantic measures. Code and data are available at https://github.com/danielle34/cavewoman.
Community
Get this paper in your agent:
hf papers read 2606.24083 curl -LsSf https://hf.co/cli/install.sh | bash Models citing this paper 0
No model linking this paper
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 0
No Space linking this paper
Collections including this paper 0
No Collection including this paper
More from Hugging Face Daily Papers
-
COrigami: An AI Pipeline for Co-Designing Flat-Foldable Visually Recognisable Origami
Jun 27
-
Fast LeWorldModel
Jun 27
-
ABACUS: Adapting Unified Foundation Model for Bridging Image Count Understanding and Generation
Jun 27
-
Neglected Free Lunch from Post-training: Progress Advantage for LLM Agents
Jun 26
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.