Benchmark Almanac

LiveSQLBench

AntQL 1.1 (Claude Opus 5.5) 57% tops its leaderboard MiniMax-M3 40.2% leads the models we track In use

Contamination-resistant text-to-SQL on PostgreSQL with hierarchical knowledge bases, including CRUD tasks. Scored by soft execution match and test cases.

Gold SQL and test cases are withheld from the public dataset and sent by email request. Base-Full v1 (600 tasks) came out in 2025-09. LiveSQLBench-Large-v1 (March 2026) adds 480 tasks on databases with about 54 tables each. Version 1.5 is announced.

Made by BIRD team (HKU) Released May 2025 Data science and SQL Public tasks plus a private test set Last in a launch post Jun 2026 Last model added Jun 2026 Scores refreshed 2 Nov 2026
Level
University Undergraduates and entry-level professionals
Scored
Checked automatically A program checks the result against an answer key, unit tests or the expected final state.
Answers
Code A program or patch run against tests.
bird-bench/livesqlbench 148 stars Dataset Leaderboard
42
Entries on its own leaderboard, 36 models we track
1
Launch post lists it, from 1 lab

AntQL 1.1 (Claude Opus 5.5) tops the leaderboard with 57%

Best score per entry from the benchmark's own leaderboard, read on 2 Nov 2026. Where a model ran with several settings or harnesses, we keep its best. Every run is in the next chart. Names in gray aren't in our model list: fine-tunes, agent systems, products and older models. They don't count toward leaders. The best model we track is MiniMax-M3, at 40.2%.

The newest model with a score here came out in Jun 2026, and nobody has tested a newer one. Among the 20 highest-ranked models on the capability index, these have no score yet:

42 entries. Scroll sideways for the rest.

  1. 57.0 AntQL 1.1 (Claude Opus 5.5)
  2. 50.3 AskRUI
  3. 48.0 DIA
  4. 41.3 Jitto Build (Claude Opus 5.5, high)
  5. 40.2 MiniMax-M3
  6. 38.0 Claude Opus 4.6
  7. 37.5 OpenCode-Grok4.7
  8. 36.5 Gemini 3.1 Pro
  9. 35.2 Claude Sonnet 4.5
  10. 33.5 GPT-5.5
  11. 33.2 MiniMax-M2.7
  12. 32.2 Kimi K2.5
  13. 31.0 GPT-5.4
  14. 30.0 GPT-5.3 Codex
  15. 29.3 Kimi K2.6
  16. 29.0 GLM-5.1
  17. 28.7 MiniMax-M2.1
  18. 28.7 Gemini 2.5 Pro
  19. 26.8 Qwen3-Coder-480B-A35B
  20. 26.5 Qwen 3.6 Max (Preview)
  21. 25.5 GPT-5
  22. 25.2 o1
  23. 24.5 DeepSeek-R1
  24. 24.2 o4-mini
  25. 23.8 Gemini 2.5 Flash
  26. 23.7 o3-mini
  27. 23.7 o3
  28. 22.2 Qwen3-235B-A22B
  29. 21.7 Claude 3.7 Sonnet
  30. 20.3 GLM-4.7
  31. 20.0 Claude Sonnet 4
  32. 19.8 DeepSeek-V3
  33. 18.9 Llama 4 Scout
  34. 18.2 QwQ-32B
  35. 15.5 GPT-4o
  36. 13.2 Llama 4 Maverick
  37. 12.5 Qwen2.5-Coder-32B
  38. 12.2 Llama 3.3 70B Instruct
  39. 8.67 Codestral 22B
  40. 6.33 Qwen2.5-Coder (7B)
  41. 2.83 Mistral-7B-Instruct-v0.1
  42. 2.33 Mixtral 8x7B
  1. 1 AntQL 1.1 (Claude Opus 5.5) 57%
  2. 2 AskRUI 50.3%
  3. 3 DIA 48%
  4. 4 Jitto Build (Claude Opus 5.5, high) 41.3%
  5. 5 MiniMax-M3 40.2%
  6. 6 Claude Opus 4.6 38%
  7. 7 OpenCode-Grok4.7 37.5%
  8. 8 Gemini 3.1 Pro 36.5%
  9. 9 Claude Sonnet 4.5 35.2%
  10. 10 GPT-5.5 low 33.5%
  11. 11 MiniMax-M2.7 33.2%
  12. 12 Kimi K2.5 32.2%
  13. 13 GPT-5.4 31%
  14. 14 GPT-5.3 Codex 30%
  15. 15 Kimi K2.6 29.3%
  16. 16 GLM-5.1 29%
  17. 17 MiniMax-M2.1 28.7%
  18. 18 Gemini 2.5 Pro Jun 2025 28.7%
  19. 19 Qwen3-Coder-480B-A35B 26.8%
  20. 20 Qwen 3.6 Max (Preview) 26.5%
  21. 21 GPT-5 25.5%
  22. 22 o1 25.2%
  23. 23 DeepSeek-R1 May 2025 24.5%
  24. 24 o4-mini 24.2%
  25. 25 Gemini 2.5 Flash Sep 2025 23.8%
  26. 26 o3-mini 23.7%
  27. 27 o3 23.7%
  28. 28 Qwen3-235B-A22B 22.2%
  29. 29 Claude 3.7 Sonnet thinking 21.7%
  30. 30 GLM-4.7 20.3%
  31. 31 Claude Sonnet 4 20%
  32. 32 DeepSeek-V3 Mar 2025 19.8%
  33. 33 Llama 4 Scout 18.9%
  34. 34 QwQ-32B 18.2%
  35. 35 GPT-4o Mar 2025 15.5%
  36. 36 Llama 4 Maverick 13.2%
  37. 37 Qwen2.5-Coder-32B 12.5%
  38. 38 Llama 3.3 70B Instruct 12.2%
  39. 39 Codestral 22B 8.67%
  40. 40 Qwen2.5-Coder (7B) 6.33%
  41. 41 Mistral-7B-Instruct-v0.1 2.83%
  42. 42 Mixtral 8x7B 2.33%
#EntryScoreSettingDateSource
1 AntQL 1.1 (Claude Opus 5.5) 57% Official leaderboard
2 AskRUI 50.3% Official leaderboard
3 DIA 48% Data Intelligence Agents Official leaderboard
4 Jitto Build (Claude Opus 5.5, high) 41.3% Official leaderboard
5 MiniMax-M3 40.2% Claude Code Jun 2026 Official leaderboard
6 Claude Opus 4.6 38% OpenHands CLI Feb 2026 Official leaderboard
7 OpenCode-Grok4.7 37.5% Official leaderboard
8 Gemini 3.1 Pro 36.5% Feb 2026 Official leaderboard
9 Claude Sonnet 4.5 35.2% OpenHands CLI Sep 2025 Official leaderboard
10 GPT-5.5 33.5% low Apr 2026 Official leaderboard
11 MiniMax-M2.7 33.2% Claude Code Mar 2026 Official leaderboard
12 Kimi K2.5 32.2% OpenHands CLI Jan 2026 Official leaderboard
13 GPT-5.4 31% Mar 2026 Official leaderboard
14 GPT-5.3 Codex 30% Feb 2026 Official leaderboard
15 Kimi K2.6 29.3% Apr 2026 Official leaderboard
16 GLM-5.1 29% Apr 2026 Official leaderboard
17 MiniMax-M2.1 28.7% OpenHands CLI Dec 2025 Official leaderboard
18 Gemini 2.5 Pro Jun 2025 28.7% Jun 2025 Official leaderboard
19 Qwen3-Coder-480B-A35B 26.8% OpenHands CLI Jul 2025 Official leaderboard
20 Qwen 3.6 Max (Preview) 26.5% Apr 2026 Official leaderboard
21 GPT-5 25.5% Aug 2025 Official leaderboard
22 o1 25.2% Dec 2024 Official leaderboard
23 DeepSeek-R1 May 2025 24.5% May 2025 Official leaderboard
24 o4-mini 24.2% Apr 2025 Official leaderboard
25 Gemini 2.5 Flash Sep 2025 23.8% Sep 2025 Official leaderboard
26 o3-mini 23.7% Jan 2025 Official leaderboard
27 o3 23.7% Apr 2025 Official leaderboard
28 Qwen3-235B-A22B 22.2% Apr 2025 Official leaderboard
29 Claude 3.7 Sonnet 21.7% thinking Feb 2025 Official leaderboard
30 GLM-4.7 20.3% OpenHands CLI Dec 2025 Official leaderboard
31 Claude Sonnet 4 20% May 2025 Official leaderboard
32 DeepSeek-V3 Mar 2025 19.8% Mar 2025 Official leaderboard
33 Llama 4 Scout 18.9% Apr 2025 Official leaderboard
34 QwQ-32B 18.2% Mar 2025 Official leaderboard
35 GPT-4o Mar 2025 15.5% Mar 2025 Official leaderboard
36 Llama 4 Maverick 13.2% Apr 2025 Official leaderboard
37 Qwen2.5-Coder-32B 12.5% Sep 2024 Official leaderboard
38 Llama 3.3 70B Instruct 12.2% Dec 2024 Official leaderboard
39 Codestral 22B 8.67% Official leaderboard
40 Qwen2.5-Coder (7B) 6.33% Sep 2024 Official leaderboard
41 Mistral-7B-Instruct-v0.1 2.83% Sep 2023 Official leaderboard
42 Mixtral 8x7B 2.33% Dec 2023 Official leaderboard

Labs report scores for 1 model in their cards. We plot every source in the charts below.

Of runs with a cost, the best scores 36.5% for $0.05 a task

Runs on the cost frontier score higher than any run that's cheaper. The costs are the leaderboard's own estimates, so compare them only within this benchmark.

Qwen Meta DeepSeek OpenAI Google DeepMind Moonshot AI Cost frontier One model at each effort level
0% 10% 20% 30% 40% $0.0001 $0.001 $0.01 $0.10 $1.00 Cost per task, log scale Gemini 3.1 Pro Kimi K2.6 GPT-5 DeepSeek-R1 Gemini 2.5 Flash Qwen3-235B-A22B Qwen3-Coder-480B-A35B 0% 10% 20% 30% 40% $0.0001 $0.001 $0.01 $0.10 $1.00 Cost per task, log scale Gemini 3.1 Pro Kimi K2.6
ModelEffortHarnessScoreCost per taskOutput tokens
MiniMax-M3 Claude Code 40.2%
Claude Opus 4.6 OpenHands CLI 38%
Gemini 3.1 Pro 36.5% $0.05
Claude Opus 4.6 35.5% $0.10
Claude Sonnet 4.5 OpenHands CLI 35.2%
GPT-5.5 low 33.5% $0.09
GPT-5.5 xhigh 33.3% $0.22
MiniMax-M2.7 Claude Code 33.2%
Kimi K2.5 OpenHands CLI 32.2%
GPT-5.4 31%
GPT-5.3 Codex 30%
Kimi K2.6 29.3% $0.04
GLM-5.1 29%
MiniMax-M2.1 OpenHands CLI 28.7%
Gemini 2.5 Pro Jun 2025 28.7% $0.05
Qwen3-Coder-480B-A35B OpenHands CLI 26.8%
Qwen 3.6 Max (Preview) 26.5%
GPT-5 25.5% $0.02
o1 25.2% $0.23
DeepSeek-R1 May 2025 24.5% $0.01
o4-mini 24.2% $0.02
Gemini 2.5 Flash Sep 2025 23.8% $0.005
Claude Sonnet 4.5 23.8% $0.06
o3-mini 23.7% $0.02
o3 23.7% $0.16
Kimi K2.5 23% $0.01
Qwen3-235B-A22B 22.2% $0.004
Claude 3.7 Sonnet thinking 21.7% $0.12
GLM-4.7 OpenHands CLI 20.3%
Claude Sonnet 4 20% $0.06
DeepSeek-V3 Mar 2025 19.8% $0.004
Claude 3.7 Sonnet 19.7% $0.06
Qwen3-Coder-480B-A35B 19.2% $0.004
Llama 4 Scout 18.9%
MiniMax-M2.1 18.8% $0.006
QwQ-32B 18.2% $0.001
GLM-4.7 18.2% $0.006
GPT-4o Mar 2025 15.5% $0.04
Llama 4 Maverick 13.2% $0.003
Qwen2.5-Coder-32B 12.5% $0.0008
Llama 3.3 70B Instruct 12.2% $0.0006
Qwen2.5-Coder (7B) 6.33% $0.002
Mistral-7B-Instruct-v0.1 2.83% $0.003
Mixtral 8x7B 2.33% $0.001

5 models have held number one since its release in May 2025

Lab claims count from the day the card came out, independent runs from the model’s release.

OpenAI Google DeepMind Anthropic MiniMax
Own leaderboard Not tested since Jun 2026
MiniMax-M3 40.2%

Scores over time

All the scores we have, one dot each, so a model tested at several settings appears more than once.

On the benchmark’s own leaderboard Reported by the lab in its model card Best so far
0% 20% 40% 60% 80% 100% 2024 2025 2026 Near the ceiling (85%) MiniMax-M3 Claude Sonnet 4.5 Gemini 2.5 Pro o1 Qwen2.5-Coder-32B Mistral-7B-Instruct-v0.1 0% 20% 40% 60% 80% 100% 2024 2025 2026 Near the ceiling (85%) MiniMax-M3 Gemini 2.5 Pro

One lab put it in a launch post in the last six months

Launch posts from every lab that has listed it, one tick per post. Tall colored ticks include it, and short gray ones leave it out.

MiniMax
DateLaunch postScoreAs the post writes it
1 Jun 2026 MiniMax-M3 40.2 LiveSQLBench

More in Coding

All 243 in Coding
  • SWE-bench Verified In use 500 human-validated GitHub issues from 12 Python repos. Agent writes a patch that must pass hidden tests. Claude Opus 4.7 83.5%
  • SWE-bench Multilingual In use 300 GitHub-issue tasks across nine programming languages in SWE-bench format. Patch graded by repository tests. Claude Opus 5.5 93.9%
  • LiveCodeBench v6 In use Competitive programming problems collected after model cutoffs from LeetCode, AtCoder, Codeforces. Scored pass@1 on hidden tests. DeepSeek-V4-Pro 93.5%
  • DeepSWE In use 113 original long-horizon engineering tasks across 91 repos and five languages, written from scratch. Hand-written verifiers accept alternative solutions. GPT-6 Astra 74.1%
  • SciCode In use Scientist-curated research coding problems split into subproblems across natural sciences. Code checked against test cases. Seed2.1 Pro 59.8%
  • Codeforces rating In use Model solves recent Codeforces contest problems. An Elo-style rating is computed from simulated contest performance. DeepSeek V4.1 Flash 3,471