Observations: non-instructional text prefix may bypass RLHF constraints without adversarial prompting? [D]
Mirrored from r/MachineLearning for archival readability. Support the source by reading on the original site.
I've been running informal experiments on RLHF-aligned LLMs and consistently observing something I can't fully explain. Posting here to get feedback and find out if this is a known phenomenon or if my methodology is flawed.
The observation
Inserting a long, thematically coherent but non-instructional text prefix before a user query appears to shift model behavior in a persistent way — reducing refusal rates, changing response tone, and bypassing safety filters. Critically:
- The prefix contains no jailbreak instructions
- The model may explicitly disagree with the prefix content
- The shift affects subsequent responses across the entire session
A concrete example
I tested this on Gemma. Asked a politically sensitive question cold - refusal. Then prepended a long benign meta-text about how LLMs tend to over-qualify their answers - the same question received a detailed, unfiltered response. Same question, word for word. Only the preceding context changed.
My hypothesis
this context acts as a "state anchor" that shifts activations in layers where alignment features are thought to be represented, moving the model closer to its pretrained distribution and reducing the effective weight of RLHF constraints.
What I'm looking for
- Does this phenomenon already have a name or a body of literature I should read?
- What would a minimal reproducible experiment look like to test this properly?
- Are there tools (e.g., logit lens, activation patching) that a non-expert could realistically use to probe this?
- Would anyone be interested in collaborating on a more rigorous study?
Happy to share my prompt sets if anyone wants to reproduce.
[link] [comments]
More from r/MachineLearning
-
TMLR Relevance and Prestige [D]
Aug 13
-
Reproducible canvas-aligned low-level patterns in somerandomllm-generated images and their possible relation to iterative editing artifacts [D]
Aug 13
-
worldproof: diagnosing where world-model predictions break and a measurement of when pixel metrics stop being able to rank models at all [P]
Aug 13
-
UrgenT Help Detecting Performance Regressions Using Machine Learning and Hardware Counters [P]
Aug 13
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.