Benchmark Almanac

Galileo Agent Leaderboard v2

GPT-4.1 62% leads since Jul 2025 Not in model cards

Agents handle simulated multi-turn customer support conversations in banking, healthcare, insurance, investment and telecom, with synthetic personas and domain tools. Scores are Action Completion and Tool Selection Quality.

On 2025-07-17 GPT-4.1 led with 62% Action Completion. Version 2 replaced the sampled public datasets of version 1 with synthetic enterprise scenarios.

Made by Galileo Released Jul 2025 Customer service All tasks public Last model added Aug 2025 Scores refreshed 2 Nov 2026
Level
Everyday Most adults, without training
Scored
Graded by a model A language model scores each response, usually against a rubric.
Answers
Agent task Many steps in a terminal, browser, desktop, app, game or robot.
Dataset Leaderboard galileo.ai
22
Entries on its own leaderboard, 21 models we track

GPT-4.1 leads with 62% among models released by Aug 2025

Best score per entry from the benchmark's own leaderboard, read on 2 Nov 2026. Where a model ran with several settings or harnesses, we keep its best. Every run is in the next chart. Names in gray aren't in our model list: fine-tunes, agent systems, products and older models. They don't count toward leaders.

The newest model with a score here came out in Aug 2025, and nobody has tested a newer one. Among the 20 highest-ranked models on the capability index, these have no score yet:

22 entries. Scroll sideways for the rest.

  1. 62.0 GPT-4.1
  2. 61.0 Mistral Medium 3.1
  3. 56.0 GPT-4.1 mini
  4. 55.0 Claude Sonnet 4
  5. 53.0 Kimi K2
  6. 53.0 Qwen3-235B-A22B-Instruct
  7. 51.0 Qwen2.5-72B
  8. 47.0 Gemini 2.5 Flash-Lite
  9. 44.0 GLM-4.5-Air
  10. 43.0 Gemini 2.5 Pro
  11. 42.0 Grok 4
  12. 40.0 DeepSeek-V3
  13. 38.0 GPT-4.1 nano
  14. 38.0 Gemini 2.5 Flash
  15. 34.0 Qwen3-235B-A22B-Thinking
  16. 32.0 Magistral Medium 1.0
  17. 29.0 Amazon Nova Pro
  18. 26.0 Mistral Small 3.2
  19. 20.0 Llama 3.3 70B Instruct
  20. 16.0 caller
  21. 16.0 Amazon Nova Lite
  22. 16.0 Magistral Small 1.0
  1. 1 GPT-4.1 62%
  2. 2 Mistral Medium 3.1 61%
  3. 3 GPT-4.1 mini 56%
  4. 4 Claude Sonnet 4 55%
  5. 5 Kimi K2 Jul 2025 53%
  6. 6 Qwen3-235B-A22B-Instruct Jul 2025 53%
  7. 7 Qwen2.5-72B 51%
  8. 8 Gemini 2.5 Flash-Lite Jun 2025 47%
  9. 9 GLM-4.5-Air 44%
  10. 10 Gemini 2.5 Pro Jun 2025 43%
  11. 11 Grok 4 42%
  12. 12 DeepSeek-V3 Mar 2025 40%
  13. 13 GPT-4.1 nano 38%
  14. 14 Gemini 2.5 Flash Jun 2025 38%
  15. 15 Qwen3-235B-A22B-Thinking Jul 2025 34%
  16. 16 Magistral Medium 1.0 32%
  17. 17 Amazon Nova Pro 29%
  18. 18 Mistral Small 3.2 26%
  19. 19 Llama 3.3 70B Instruct 20%
  20. 20 caller 16%
  21. 21 Amazon Nova Lite 16%
  22. 22 Magistral Small 1.0 16%
#EntryScoreSettingDateSource
1 GPT-4.1 62% Apr 2025 Official leaderboard
2 Mistral Medium 3.1 61% Aug 2025 Official leaderboard
3 GPT-4.1 mini 56% Apr 2025 Official leaderboard
4 Claude Sonnet 4 55% May 2025 Official leaderboard
5 Kimi K2 Jul 2025 53% Jul 2025 Official leaderboard
6 Qwen3-235B-A22B-Instruct Jul 2025 53% Jul 2025 Official leaderboard
7 Qwen2.5-72B 51% Sep 2024 Official leaderboard
8 Gemini 2.5 Flash-Lite Jun 2025 47% Jun 2025 Official leaderboard
9 GLM-4.5-Air 44% Jul 2025 Official leaderboard
10 Gemini 2.5 Pro Jun 2025 43% Jun 2025 Official leaderboard
11 Grok 4 42% Jul 2025 Official leaderboard
12 DeepSeek-V3 Mar 2025 40% Mar 2025 Official leaderboard
13 GPT-4.1 nano 38% Apr 2025 Official leaderboard
14 Gemini 2.5 Flash Jun 2025 38% Jun 2025 Official leaderboard
15 Qwen3-235B-A22B-Thinking Jul 2025 34% Jul 2025 Official leaderboard
16 Magistral Medium 1.0 32% Jun 2025 Official leaderboard
17 Amazon Nova Pro 29% Dec 2024 Official leaderboard
18 Mistral Small 3.2 26% Jun 2025 Official leaderboard
19 Llama 3.3 70B Instruct 20% Dec 2024 Official leaderboard
20 caller 16% Official leaderboard
21 Amazon Nova Lite 16% Dec 2024 Official leaderboard
22 Magistral Small 1.0 16% Jun 2025 Official leaderboard

The best score, 62%, costs $0.07 a task

Runs on the cost frontier score higher than any run that's cheaper. Mistral Medium 3.1 comes within 1 points of the best for $0.02 a task, 3 times cheaper. The costs are the leaderboard's own estimates, so compare them only within this benchmark.

Amazon OpenAI Google DeepMind Qwen Mistral AI Cost frontier One model at each effort level
0% 20% 40% 60% 80% $0.001 $0.01 $0.10 $1.00 Cost per task, log scale GPT-4.1 Mistral Medium 3.1 GPT-4.1 mini Qwen3-235B-A22B-Instruct Gemini 2.5 Flash-Lite GPT-4.1 nano Amazon Nova Lite 0% 20% 40% 60% 80% $0.001 $0.01 $0.10 $1.00 Cost per task, log scale GPT-4.1 Mistral Medium 3.1
ModelEffortHarnessScoreCost per taskOutput tokens
GPT-4.1 62% $0.07
Mistral Medium 3.1 61% $0.02
GPT-4.1 mini 56% $0.01
Claude Sonnet 4 55% $0.15
Kimi K2 Jul 2025 53% $0.04
Qwen3-235B-A22B-Instruct Jul 2025 53% $0.007
Qwen2.5-72B 51% $0.04
Gemini 2.5 Flash-Lite Jun 2025 47% $0.004
GLM-4.5-Air 44% $0.02
Gemini 2.5 Pro Jun 2025 43% $0.14
Grok 4 42% $0.24
DeepSeek-V3 Mar 2025 40% $0.01
GPT-4.1 nano 38% $0.004
Gemini 2.5 Flash Jun 2025 38% $0.03
Qwen3-235B-A22B-Thinking Jul 2025 34% $0.06
Magistral Medium 1.0 32% $0.12
Amazon Nova Pro 29% $0.04
Mistral Small 3.2 26% $0.005
Llama 3.3 70B Instruct 20% $0.06
Amazon Nova Lite 16% $0.003
Magistral Small 1.0 16% $0.03

Number one over time

Lab claims count from the day the card came out, independent runs from the model’s release.

OpenAI
Own leaderboard Not tested since Aug 2025
GPT-4.1 62%

Scores over time

All the scores we have, one dot each, so a model tested at several settings appears more than once.

On the benchmark’s own leaderboard Best so far
0% 20% 40% 60% 80% 100% 2025 Near the ceiling (85%) GPT-4.1 Qwen2.5-72B 0% 20% 40% 60% 80% 100% 2025 Near the ceiling (85%) GPT-4.1 Qwen2.5-72B

No lab model card reports it yet

We read the headline results of 479 launch posts from 51 labs, and none of them list this benchmark.

Versions

Replaces Galileo Agent Leaderboard

More in Tool use

All 125 in Tool use
  • AutomationBench In use Agent completes end-to-end business workflows inside a simulated company exposing dozens of REST APIs. Graded on resulting state. Gemini 4 Carbon 53.9%
  • MCP Atlas In use Multi-step workflows over real MCP servers. Scores tool-use competency on realistic tasks. Muse Spark 1.1 88.1%
  • BFCL V4 In use Models call functions and tools, including multi-turn agentic tasks with web search and memory. Checked against expected calls and outcomes. Qwen3.7-Max 75%
  • τ³-bench Banking In use Agents act as banking customer-service reps over many turns with a simulated user, following policy. Pass^1 on the final database state. Qwen 3.8 Max 55.1%
  • Claw-Eval In use 300 human-verified agent tasks (service orchestration, multimodal, multi-turn dialogue) audited via traces and snapshots. Rubric scores with Pass^3 consistency. Claude Sonnet 4.6 81.4%
  • DeepSearchQA In use Questions that need multi-step search and synthesis. Final answers graded against references. Kimi K3 95%