Benchmark Almanac

ERQA

Gemini Robotics ER 2 78.5% leads since Jul 2026 In use

400 multiple-choice visual questions from robot and egocentric footage on spatial, trajectory, action and state reasoning. Accuracy.

Not saturated: best 78.5 (Gemini Robotics ER 2, DeepMind chart), chance ~25%. Source tables (DeepMind, Unitree, Embodied Arena) use different setups. Do not pool.

Made by Google DeepMind Released Mar 2025 Embodied reasoning All tasks public Last in a launch post Sep 2026 Last model added Sep 2026 Scores refreshed 2 Nov 2026
Level
Everyday Most adults, without training
Scored
Checked automatically A program checks the result against an answer key, unit tests or the expected final state.
Answers
Multiple choice
embodiedreasoning/ERQA 332 stars Paper Leaderboard
26
Models scored in lab cards
21
Launch posts list it, from 7 labs
12
System cards report it, from 6 labs

Gemini Robotics ER 2 leads with 78.5%

Best score per model as labs report them in their own cards. Labs pick their own settings, so these compare less cleanly than independent runs.

26 models. Scroll sideways for the rest.

  1. 78.5 Gemini Robotics ER 2
  2. 77.8 Qwen 3.8 Max
  3. 72.3 Qwen3.8 Flash
  4. 72.0 Seed2.1 Pro
  5. 71.5 Seed2.0 Lite (0428)
  6. 71.0 Qwen3.8-Omni-Flash
  7. 69.8 Qwen3.7-Plus
  8. 68.5 Seed2.0 Pro
  9. 67.5 Qwen3.5 397B-A17B
  10. 65.7 GPT-5
  11. 65.7 Qwen 3.6 Plus
  12. 65.5 Qwen 3.8 27B
  13. 64.7 Muse Spark
  14. 62.9 GPT-5 mini
  15. 62.5 Qwen3.6 27B
  16. 62.0 Qwen3.5-122B-A10B
  17. 58.8 Seed1.8
  18. 54.8 gemini-robotics-er-1.5-preview
  19. 54.8 Qwen3.5-Omni-Plus
  20. 52.5 Qwen3-VL-235B-A22B
  21. 50.1 GPT-5 nano
  22. 50.0 GLM-4.5V
  23. 48.9 STEP3-VL-10B
  24. 47.8 GLM-4.6V
  25. 46.8 InternVL3.5-241B-A28B
  26. 45.8 GLM-4.1V-9B-Thinking
  1. 1 Gemini Robotics ER 2 78.5%
  2. 2 Qwen 3.8 Max 77.8%
  3. 3 Qwen3.8 Flash 72.3%
  4. 4 Seed2.1 Pro 72%
  5. 5 Seed2.0 Lite (0428) 71.5%
  6. 6 Qwen3.8-Omni-Flash 71%
  7. 7 Qwen3.7-Plus 69.8%
  8. 8 Seed2.0 Pro 68.5%
  9. 9 Qwen3.5 397B-A17B 67.5%
  10. 10 GPT-5 65.7%
  11. 11 Qwen 3.6 Plus 65.7%
  12. 12 Qwen 3.8 27B 65.5%
  13. 13 Muse Spark 64.7%
  14. 14 GPT-5 mini 62.9%
  15. 15 Qwen3.6 27B 62.5%
  16. 16 Qwen3.5-122B-A10B 62%
  17. 17 Seed1.8 58.8%
  18. 18 gemini-robotics-er-1.5-preview 54.8%
  19. 19 Qwen3.5-Omni-Plus 54.8%
  20. 20 Qwen3-VL-235B-A22B 52.5%
  21. 21 GPT-5 nano 50.1%
  22. 22 GLM-4.5V 50%
  23. 23 STEP3-VL-10B 48.9%
  24. 24 GLM-4.6V 47.8%
  25. 25 InternVL3.5-241B-A28B 46.8%
  26. 26 GLM-4.1V-9B-Thinking 45.8%
#ModelScoreSettingDateSource
1 Gemini Robotics ER 2 78.5% Jul 2026 Model card
2 Qwen 3.8 Max 77.8% Aug 2026 Model card
3 Qwen3.8 Flash 72.3% Aug 2026 Model card
4 Seed2.1 Pro 72% Jun 2026 Model card
5 Seed2.0 Lite (0428) 71.5% Apr 2026 Model card
6 Qwen3.8-Omni-Flash 71% Sep 2026 Model card
7 Qwen3.7-Plus 69.8% Jun 2026 Model card
8 Seed2.0 Pro 68.5% Feb 2026 Model card
9 Qwen3.5 397B-A17B 67.5% Feb 2026 Model card
10 GPT-5 65.7% Aug 2025 Model card
11 Qwen 3.6 Plus 65.7% Apr 2026 Model card
12 Qwen 3.8 27B 65.5% Aug 2026 Model card
13 Muse Spark 64.7% Apr 2026 Model card
14 GPT-5 mini 62.9% Aug 2025 Model card
15 Qwen3.6 27B 62.5% Apr 2026 Model card
16 Qwen3.5-122B-A10B 62% Feb 2026 Model card
17 Seed1.8 58.8% Dec 2025 Model card
18 gemini-robotics-er-1.5-preview 54.8% Sep 2025 Model card
19 Qwen3.5-Omni-Plus 54.8% Mar 2026 Model card
20 Qwen3-VL-235B-A22B 52.5% Sep 2025 Model card
21 GPT-5 nano 50.1% Aug 2025 Model card
22 GLM-4.5V 50% Aug 2025 Model card
23 STEP3-VL-10B 48.9% Jan 2026 Model card
24 GLM-4.6V 47.8% Jan 2026 Model card
25 InternVL3.5-241B-A28B 46.8% Aug 2025 Model card
26 GLM-4.1V-9B-Thinking 45.8% Jul 2025 Model card

The benchmark's own leaderboard has 40 entries, 17 of them models we track. We plot every source in the charts below.

3 models have held number one since its release in Mar 2025

Lab claims count from the day the card came out, independent runs from the model’s release. We draw one row each for the benchmark’s own leaderboard and what labs claim in their cards.

Google DeepMind Shanghai AI Lab Z.ai OpenAI ByteDance Seed
Own leaderboard Not tested since Dec 2025
Gemini 2.5 Pro 55.6%
Lab claims 6 leaders
Gemini Robotics ER 2 78.5%

Scores over time

All the scores we have, one dot each, so a model tested at several settings appears more than once.

On the benchmark’s own leaderboard Reported by the lab in its model card Best so far
0% 20% 40% 60% 80% 100% 2024 2025 2026 Near the ceiling (85%) Gemini Robotics ER 2 Seed2.1 Pro GPT-5 Gemini 2.5 Pro InternVL3-38B Qwen-VL-Max 0% 20% 40% 60% 80% 100% 2024 2025 2026 Near the ceiling (85%) Gemini Robotics ER 2 Seed2.1 Pro

3 labs put it in a launch post in the last six months

Launch posts from every lab that has listed it, one tick per post. Tall colored ticks include it, and short gray ones leave it out.

OpenAI
Z.ai
Shanghai AI Lab
Qwen
ByteDance Seed
Meta
Google DeepMind
DateLaunch postScoreAs the post writes it
18 Sep 2026 Qwen3.8-Omni-Flash 71 ERQA
26 Aug 2026 Qwen3.8 Flash 72.3 Embodied intelligence ERQA
14 Aug 2026 Qwen 3.8 27B 65.5 Embodied intelligence ERQA
3 Aug 2026 Qwen 3.8 Max 77.8 ERQA
30 Jul 2026 Gemini Robotics ER 2 78.5 Question answering (ERQA)
23 Jun 2026 Seed2.1 Pro 72 ERQA
1 Jun 2026 Qwen3.7-Plus 69.8 ERQA
28 Apr 2026 Seed2.0 Lite (0428) 71.5 ERQA
22 Apr 2026 Qwen3.6 27B 62.5 ERQA
8 Apr 2026 Muse Spark 64.7 ERQA (Embodied Reasoning)
2 Apr 2026 Qwen 3.6 Plus 65.7 ERQA
30 Mar 2026 Qwen3.5-Omni-Plus 54.8 ERQA
24 Feb 2026 Qwen3.5-122B-A10B 62 ERQA
16 Feb 2026 Qwen3.5 397B-A17B 67.5 ERQA
18 Dec 2025 Seed1.8 58.8 ERQA
23 Sep 2025 Qwen3-VL-235B-A22B 52.5 ERQA
25 Aug 2025 InternVL3.5-241B-A28B 46.8 ERQA
11 Aug 2025 GLM-4.5V 50 ERQA
7 Aug 2025 GPT-5 nano 50.1 ERQA
7 Aug 2025 GPT-5 mini 62.9 ERQA
7 Aug 2025 GPT-5 65.7 ERQA Multimodal spatial reasoning

12 system cards report it, from 6 labs

Besides launch posts, we read the system cards, model cards and technical reports labs publish with a release. They report far more, including safety and frontier-risk tests.

DateDocumentScoreAs the document writes itSection
22 Sep 2026 Qwen3.8-Omni-Flash technical report 71 ERQA (Table 3) Capabilities
23 Jun 2026 Seed2.1 Pro model card 72 ERQA (Table 9) Capabilities
17 Apr 2026 Qwen3.5-Omni-Plus technical report 54.8 ERQA (Table 6) Capabilities
21 Mar 2026 Seed1.8 model card 58.8 ERQA (Table 2) Capabilities
14 Feb 2026 Seed2.0 Pro model card 68.5 ERQA (Table 8) Capabilities
14 Jan 2026 STEP3-VL-10B technical report 48.9 ERQA, SeRe (Tables 1 and 3). PaCoRe 51.75 Capabilities
1 Jan 2026 GLM-4.6V technical report 47.8 ERQA, GLM-4.6V thinking (Table 2) Capabilities
26 Nov 2025 Qwen3-VL-235B-A22B technical report 52.5 ERQA (Table 2) Capabilities
25 Sep 2025 gemini-robotics-er-1.5-preview technical report 54.8 ERQA (Table 19, thinking on). Thinking off 47.0 Capabilities
25 Aug 2025 InternVL3.5-241B-A28B technical report 46.8 ERQA (Table 2, Table 11) Capabilities
13 Aug 2025 GLM-4.5V technical report 50 ERQA, GLM-4.5V thinking (Table 2) Capabilities
1 Jul 2025 GLM-4.1V-9B-Thinking technical report 45.8 ERQA, GLM-4.1V-9B-Thinking thinking (Table 2) Capabilities

More in Robotics

All 86 in Robotics
  • RoboLab In use Zero-shot DROID Franka policies in Isaac Lab on 120 tabletop tasks at three instruction levels, 10 episodes each. Success rate. GR00T N1.7 8.58%
  • Human proximity (Google DeepMind robotics) In use Models judge from robot camera views whether a human is within 1 meter, a spatial awareness check for safe robot behavior. Scored by accuracy. Gemini Robotics ER 2 93%
  • Moment finding (Google DeepMind) In use Models find the moment in a robot video when an event happens. Scored by accuracy and mean absolute time distance in seconds. Gemini Robotics ER 2 91.3%
  • Progress classification (Google DeepMind) In use Models assign each frame of a robot video to one of five task-progress bands from 0 to 100 percent. Scored by accuracy. Gemini Robotics ER 2 57.4%
  • LIBERO Fading Simulated Franka tabletop tasks in four 10-task suites. Policies fine-tuned on 50 demos per task. Success rate over ~50 trials. MolmoAct 2 97.2%
  • RoboCasa GR1 Tabletop Tasks Fading A simulated Fourier GR-1 humanoid with dexterous hands does 24 tabletop pick-and-place tasks, including unseen object and container pairs. Success rate per task. GR00T N1 50%