Despite not being trained to, it turns out the Pearson correlation between a models AA Intelligence Index score and its ability to generate Base64 encoded responses is 0.91

I built Encode Bench, an open benchmark that asks a model to solve a task and return the answer as a Base64 payload.

The initial result surprised me: across the eight models with matching data in the current nine-model snapshot, Encode Bench pass rate has a Pearson correlation of 0.91 with the Artificial Analysis Intelligence Index. The correlation with its Agentic Index is 0.94.

That sounds dramatic, so the caveat belongs right next to it: this is a small, imperfect observational sample. It does not show that Base64 measures intelligence, and it does not establish causation. SimpleBench is a useful counterexample: its correlation with Encode Bench is only 0.23, although that comparison has just four overlapping models.

The idea came from an asymmetry I kept seeing: models could often interpret Base64 in a prompt, but some struggled to produce Base64 that decoded into the exact artifact requested. Generating the final payload requires the model to:

1. solve the underlying problem;
2. preserve the answer exactly;
3. encode it correctly; and
4. follow a very narrow output contract.

A failure at any link breaks the artifact, so this may be a crude test of multi-step reliability. Or it may mostly reflect tokenizer behavior, training data, post-training, reasoning limits, or provider routing. The current benchmark cannot separate those explanations.

The scored battery contains 24 deterministic tasks across encoding fidelity, instruction following, arithmetic, logic, code reasoning, and structured data. Each task is run three times, giving 72 scored trials per model. Missing trials, provider failures, invalid Base64, output-cap failures, and Base64 containing the wrong answer all count as failures. A Base64-encoded PNG prompt is included only as a subjective showcase and never enters the score.

Current results:

• GPT-5.6 Sol — 70/72 (97.2%)
• Kimi K3 — 63/72 (87.5%)
• Claude Sonnet 5 — 49/72 (68.1%)
• Gemini 3.5 Flash — 46/72 (63.9%)
• DeepSeek V4 Flash — 43/72 (59.7%)
• Hy3 (free) — 38/72 (52.8%)
• Laguna S 2.1 (free) — 31/72 (43.1%)
• Nemotron 3 Nano 30B A3B (free) — 23/72 (31.9%)
• Gemma 4 26B A4B IT (free) — 17/72 (23.6%)

One result I did not expect: raw encoding-fidelity tasks were the hardest category at 35.2%, while code reasoning was the easiest at 74.1%. Many failures were not malformed Base64 at all—the payload decoded successfully but contained the wrong answer. The score is therefore mixing reasoning, exactness, encoding, endpoint reliability, and inference limits. That mixture may help explain the correlation, but it is also the strongest reason not to over-interpret it.

The biggest missing experiment is a matched plain-text control battery with the Base64 requirement removed. I would also like to test hexadecimal and matched random strings.

Interactive results and per-trial outputs:

https://arvidsu.github.io/encode_bench/

Source, prompts, model configs, and scoring code:

https://github.com/ArvidSU/encode_bench

I would be interested in this community's read: is encoded generation exposing a real generalization gap, or mostly a tokenizer/training artifact? And as benchmarks like this enter training data, does the signal improve or simply stop meaning what it meant before?