r/MachineLearning · · 1 min read

Observations: non-instructional text prefix may bypass RLHF constraints without adversarial prompting? [D]

Mirrored from r/MachineLearning for archival readability. Support the source by reading on the original site.

I've been running informal experiments on RLHF-aligned LLMs and consistently observing something I can't fully explain. Posting here to get feedback and find out if this is a known phenomenon or if my methodology is flawed.

The observation

Inserting a long, thematically coherent but non-instructional text prefix before a user query appears to shift model behavior in a persistent way — reducing refusal rates, changing response tone, and bypassing safety filters. Critically:

  • The prefix contains no jailbreak instructions
  • The model may explicitly disagree with the prefix content
  • The shift affects subsequent responses across the entire session

A concrete example

I tested this on Gemma. Asked a politically sensitive question cold - refusal. Then prepended a long benign meta-text about how LLMs tend to over-qualify their answers - the same question received a detailed, unfiltered response. Same question, word for word. Only the preceding context changed.

My hypothesis

this context acts as a "state anchor" that shifts activations in layers where alignment features are thought to be represented, moving the model closer to its pretrained distribution and reducing the effective weight of RLHF constraints.

What I'm looking for

  • Does this phenomenon already have a name or a body of literature I should read?
  • What would a minimal reproducible experiment look like to test this properly?
  • Are there tools (e.g., logit lens, activation patching) that a non-expert could realistically use to probe this?
  • Would anyone be interested in collaborating on a more rigorous study?

Happy to share my prompt sets if anyone wants to reproduce.

submitted by /u/Historical-Cod-2537
[link] [comments]

Discussion (0)

Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.

Sign in →

No comments yet. Sign in and be the first to say something.

More from r/MachineLearning