BigCodeBench (Elo) bigcodebench-elo Leaderboard
Elo-style pairwise rating across BigCodeBench tasks. Complementary to Pass@1 — captures relative strength when multiple models attempt the same prompts. Bigger differences in Elo correlate with bigger Pass@1 gaps. · Metric: Elo (higher is better)
| # | Model | Elo | Paper |
|---|---|---|---|
| 1 | DeepSeek-V3-Chat | 1216.89 | — |
| 2 | GPT-4o-2024-05-13 | 1216.72 | — |
| 3 | DeepSeek-V2-Chat (2024-06-28) | 1186.31 | — |
| 4 | DeepSeek-Coder-V2-Instruct | 1184.20 | — |
| 5 | Gemini-Exp-1114 | 1173.74 | — |
| 6 | Gemini-Exp-1206 | 1172.42 | — |
| 7 | Qwen2.5-Coder-32B-Instruct | 1168.91 | — |
| 8 | GPT-4-Turbo-2024-04-09 | 1162.95 | — |
| 9 | GPT-4o-2024-11-20 | 1156.35 | — |
| 10 | Claude-3.5-Sonnet-20240620 | 1146.48 | — |
| 11 | GPT-4-0613 | 1143.07 | — |
| 12 | Codestral-2501 | 1142.93 | — |
| 13 | Claude-3.5-Haiku-20241022 | 1142.85 | — |
| 14 | Gemini-2.0-Flash-Exp | 1142.47 | — |
| 15 | Llama-3.3-70B-Instruct | 1142.14 | — |
| 16 | GPT-4o-mini-2024-07-18 | 1141.20 | — |
| 17 | Athene-V2-Chat | 1140.81 | — |
| 18 | Claude-3-Opus-20240229 | 1132.72 | — |
| 19 | Athene-V2-Agent | 1128.42 | — |
| 20 | Hermes-2-Theta-Llama-3-70B | 1127.49 | — |
| 21 | Qwen2.5-72B-Instruct | 1125.66 | — |
| 22 | Gemini-Exp-1121 | 1123.33 | — |
| 23 | Gemini-1.5-Pro-API-0514 | 1123.08 | — |
| 24 | DeepSeek-V2.5-1210 | 1123.05 | — |
| 25 | Llama-3.1-70B-Instruct | 1122.56 | — |
| 26 | Phi-4 | 1119.78 | — |
| 27 | Claude-3.5-Sonnet-20241022 | 1112.66 | — |
| 28 | Gemini-1.5-Flash-API-0514 | 1105.38 | — |
| 29 | Llama-3-70B-Instruct | 1099.57 | — |
| 30 | Llama-3-70B-Synthia-v3.5 | 1096.57 | — |