r/LocalLLaMA · · 1 min read

I collected every single LLM coding benchmark, and computed their Intelligence Density

Mirrored from r/LocalLLaMA for archival readability. Support the source by reading on the original site.

I collected every single LLM coding benchmark, and computed their Intelligence Density

The intelligence in my context is an aggregate index, I called the Agentic Coding Index, across most relevant agentic coding benchmarks: SWE-bench Pro, DeepSWE v1.1, Terminal-Bench (v4, v3, v2.1), Code Arena Elo, and LiveCodeBench v6.

Intelligence/Parameter=Scale x (Agentic Index / Norm) ^ (Super_Linear_Exponent) / sqrt(PCount + PLowerBound)

  • Norm: sets a neutral baseline (= 50).
  • Super_Linear_Exponent: non-linear scale to avoid rewarding very small models (otherwise, small models that can barely write code would artificially dominate the leaderboard), while rewarding true autonomous mastery. Scale = 2.5354.
  • PCount: model parameter count (in Billions).
  • PLowerBound: minimum count of model parameters (regularization term, to avoid models <1B shooting up the score), =8B.

Agentic Coding Index: DeepSWE v1.1 (20%), Code Arena Elo (20%), Terminal-Bench v4.0 (15%), SWE-bench Pro (15%), Terminal-Bench v3.0 (13%), Terminal-Bench v2.1 (12%), and LiveCodeBench v6 (5%).

Data Integrity: All benchmark scores are curated from verified public and official sources (model creators, peer-reviewed evaluation reports).

submitted by /u/Informal-Trouble2183
[link] [comments]

Discussion (0)

Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.

Sign in →

No comments yet. Sign in and be the first to say something.

More from r/LocalLLaMA