Brood War Bench
Mirrored from Hacker News — AI on Front Page for archival readability. Support the source by reading on the original site.
Key takeaways
- None of the models played beyond a beginner level.
- Codex Astra is the clear leader beating all other models consistently.
- Grok models are not smart enough to play Brood War yet.
- Older models tended to play the RTS as a turn-based game, leading them to get destroyed while they were thinking. Newer models sometimes fell into the same trap, which may explain why some lower-effort settings performed better, but overall were much more cognizant of the cost of thinking.
Powered By Freestyle
Leaderboard
| Rank | System | Wins | Losses | APM | Cost / game | Win rate |
|---|---|---|---|---|---|---|
| 🥇 |  Codex Astra / xhigh | 18 | 0 | 12.6 | $10.54 | 100.0% |
| Codex Astra / medium | 16 | 2 | 17.2 | $15.11 | 88.9% | |
 Claude Fable | 15 | 3 | 12.6 | $12.24 | 83.3% | |
| Codex Astra / low | 14 | 4 | 25.7 | $21.07 | 77.8% | |
| Codex 5.6 Sol / medium | 13 | 5 | 10.1 | $5.12 | 72.2% | |
| Codex 5.6 Sol / low | 12 | 6 | 18.1 | $9.23 | 66.7% | |
| Claude Opus 5 | 12 | 6 | 10.5 | $20.78 | 66.7% | |
| Codex 5.6 Sol / xhigh | 11 | 7 | 8.0 | $3.23 | 61.1% | |
| Codex 5.6 Luna / low | 9 | 9 | 23.8 | $0.42 | 50.0% | |
| Codex 5.6 Terra / xhigh | 9 | 9 | 15.8 | $2.10 | 50.0% | |
| Codex 5.6 Terra / medium | 8 | 10 | 10.5 | $3.15 | 44.4% | |
| Codex 5.6 Terra / low | 8 | 10 | 48.3 | $4.65 | 44.4% | |
| Codex 5.6 Luna / xhigh | 7 | 11 | 5.2 | $0.16 | 38.9% | |
| Claude Sonnet | 7 | 11 | 6.2 | $8.98 | 38.9% | |
| Codex 5.6 Luna / medium | 6 | 12 | 14.8 | $0.30 | 33.3% | |
 Grok 4.6 / xhigh | 2 | 15 | 2.8 | $0.66 | 11.1% | |
| Grok 4.6 / medium | 1 | 16 | 3.2 | $0.79 | 5.6% | |
| Claude Haiku | 0 | 16 | 0.3 | $0.34 | 0.0% | |
| Grok 4.6 / low | 0 | 16 | 4.2 | $1.26 | 0.0% |
Brood War Bench started after I built a version of Brood War that you could only play through agents as an experiment to play with friends. I played it with a couple friends who did surprisingly well for people who have only played a couple Starcraft games in their lives. When I asked them why, they said they hadn't done much, they asked their agent to attack and it had built a small army and done the full attack for them. This lead me to wonder how far they can go on their own; this is my answer.
What I observed
01
Codex found cheese before it found macro
Codex's strongest recurring idea was disruption. In Protoss games it often sent a Probe across the map to attack workers or buildings. This worked shockingly well as the opposing agents often spent dozens of seconds thinking about what to do about a probe instead of doing anything else.
The same systems were much weaker at sustained production. They delayed tech, trickled one or two basic units into defended bases, and threw workers into last stands.
I also noticed Codex often created separate subagents to manage the economy, army production, and army control. They didn't communicate much with one another, so the army agent often sent each new unit straight into an attack, unaware of the larger army the other agents were planning to build.
This is a common beginner mistake: sending units in one at a time instead of waiting for a critical mass and a planned attack timing. In games where I helped direct Codex, it was much better at planning those moments and getting its subagents to work together.
The persistence was real. In G009, after losing its army and main base, Codex 5.6 Terra / medium lifted its last Command Center and moved it toward the opposite corner. It survived for another six minutes.
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.