Benchmark Almanac

VAKRA

Gemini 3 Flash 38.1% leads since March 2026
Not in model cards

Agents answer multi-step questions by calling more than 8,000 locally hosted APIs over real databases in 62 domains, along with matching document collections. Tasks need chains of 3 to 7 steps across APIs and documents. A deterministic evaluator replays tools and checks the full trajectory.

The score averages four parts: API chaining, tool selection, multi-hop and multi-turn. IBM ran five models with a ReAct agent, and one outside submission had entries only for tool selection. About 77 GitHub stars.

Made by
IBM Research
Released
March 2026
Kind
Function calling
Level
Everyday
Scored
Checked automatically
Answers
Agent task
Tasks
All public
Strongest model tested
Gemini 3 Flash #59 on the capability index
Last model added
April 2026
Scores refreshed
November 12th, 2026
6
Entries on its own leaderboard, 4 models we track

Gemini 3 Flash leads with 38.1% among models released by April 2026

Best score per entry from the benchmark's own leaderboard, read on November 12th, 2026. Where a model ran with several settings or harnesses, we keep its best. Names in gray aren't in our model list: fine-tunes, agent systems, products and older models. They don't count toward leaders.

The newest model with a score here came out in April 2026, and nobody has tested a newer one. Among the 20 highest-ranked models on the capability index, these have no score yet:

  1. 38.1 Gemini 3 Flash
  2. 34.2 gpt-oss-120b
  3. 32.8 Claude Sonnet 4.5
  4. 32.4 LLAMA-405B
  5. 28.5 Granite-4.0-h-small-32B
  6. 14.9 Qwen 3.6 35B-A3B
  1. 1 Gemini 3 Flash 38.1%
  2. 2 gpt-oss-120b 34.2%
  3. 3 Claude Sonnet 4.5 32.8%
  4. 4 LLAMA-405B 32.4%
  5. 5 Granite-4.0-h-small-32B 28.5%
  6. 6 Qwen 3.6 35B-A3B 14.9%

Source: Official VAKRA leaderboard

#EntryScoreSettingDateSource
1 Gemini 3 Flash 38.1% ReAct (Prompt) Dec 2025 Official leaderboard
2 gpt-oss-120b 34.2% ReAct (Prompt) Aug 2025 Official leaderboard
3 Claude Sonnet 4.5 32.8% ReAct (Prompt) Sep 2025 Official leaderboard
4 LLAMA-405B 32.4% ReAct (Prompt) Official leaderboard
5 Granite-4.0-h-small-32B 28.5% ReAct (Prompt) Official leaderboard
6 Qwen 3.6 35B-A3B 14.9% Juris Apr 2026 Official leaderboard

Number one over time

Lab claims count from the day the card came out, independent runs from the model’s release.

Google DeepMind
Own leaderboard Not tested since Apr 2026
Gemini 3 Flash 38.1%

Source: Official VAKRA leaderboard

Scores over time

All the scores we have, one dot each, so a model tested at several settings appears more than once.

On the benchmark’s own leaderboard Best so far
0% 20% 40% 60% 80% 100% 2026 Near the ceiling (85%) Gemini 3 Flash gpt-oss-120b 0% 20% 40% 60% 80% 100% 2026 Near the ceiling (85%) Gemini 3 Flash

Source: Official VAKRA leaderboard

No lab model card reports it yet

We read the headline results of 522 launch posts from 52 labs, and none of them list this benchmark.

More in Tool use

All 144 in Tool use
  • AutomationBench In use Agent completes end-to-end business workflows inside a simulated company exposing dozens of REST APIs. Graded on resulting state. Gemini 4 Carbon 53.9%
  • MCP Atlas In use Multi-step workflows over real MCP servers. Scores tool-use competency on realistic tasks. Gemini 4 Argon 90.7%
  • BFCL V4 In use Models call functions and tools, including multi-turn agentic tasks with web search and memory. Checked against expected calls and outcomes. Qwen3.7-Max 75%
  • τ³-bench Banking In use Agents act as banking customer-service reps over many turns with a simulated user, following policy. Pass^1 on the final database state. Qwen 3.8 Max 55.1%
  • Toolathlon-Verified In use 108 long-horizon tasks across 32 apps through about 600 MCP tools. Execution-based checkers inspect final environment state. GLM-5.3-Flash 78.4%
  • DeepSearchQA In use Questions that need multi-step search and synthesis. Final answers graded against references. Kimi K3 95%