RoboEval
MolmoAct 2 0.44 Fading Robot policies perform eight bimanual manipulation tasks with controlled variations in simulation. Scores add behavior and progress metrics, such as coordination and efficiency, to task success.
The paper ships over three thousand expert demonstrations. Version 2 of the paper came out on 2026-05-04. Ai2 reports a 0 to 1 score for MolmoAct 2.
In the authors’ words
We introduce RoboEval, a structured evaluation framework and benchmark for robotic manipulation that augments binary success with principled behavioral and outcome metrics. Existing evaluations often collapse performance into outcome counts, masking differences in execution quality and obscuring failure structure. RoboEval provides eight bimanual tasks with systematically controlled variations, more than three thousand expert demonstrations, and a modular simulation platform for reproducible experimentation. All tasks are instrumented with standardized metrics that quantify efficiency, coordination, and safety/stability, as well as outcome measures that trace stagewise progress and localize failure modes. Through extensive experiments with state-of-the-art visuomotor policies, we validate these metrics by analyzing their stability under variation, discriminative power across policies with similar success rates, and correlation with task success. Project Page: https://robo-eval.github.io
- Level
- Everyday Most adults, without training
- Scored
- Checked automatically A program checks the result against an answer key, unit tests or the expected final state.
- Answers
- Agent task Many steps in a terminal, browser, desktop, app, game or robot.
MolmoAct 2 leads with 0.44
Best score per model as labs report them in their own cards. Labs pick their own settings, so these compare less cleanly than independent runs.
- 1 MolmoAct 2 0.44
| # | Model | Score | Setting | Date | Source |
|---|---|---|---|---|---|
| 1 | MolmoAct 2 | 0.44 | May 2026 | Model card |
Scores over time
All the scores we have, one dot each, so a model tested at several settings appears more than once.
Labs headline it less often now, last in May 2026
Launch posts from every lab that has listed it, one tick per post. Tall colored ticks include it, and short gray ones leave it out.
| Date | Launch post | Score | As the post writes it |
|---|---|---|---|
| 5 May 2026 | MolmoAct 2 | 0.443 | RoboEval (0 to 1, higher is better) |
One system card reports it
Besides launch posts, we read the system cards, model cards and technical reports labs publish with a release. They report far more, including safety and frontier-risk tests.
| Date | Document | Score | As the document writes it | Section |
|---|---|---|---|---|
| 4 May 2026 | MolmoAct 2 technical report | 44.3 | RoboEval success rate averaged over 8 bimanual tasks, fine-tuned (6.3 text and Figure 6) | Capabilities |
More in Robotics
- ERQA In use 400 multiple-choice visual questions from robot and egocentric footage on spatial, trajectory, action and state reasoning. Accuracy.
Gemini Robotics ER 2 78.5% - RoboLab In use Zero-shot DROID Franka policies in Isaac Lab on 120 tabletop tasks at three instruction levels, 10 episodes each. Success rate.
GR00T N1.7 8.58% - Human proximity (Google DeepMind robotics) In use Models judge from robot camera views whether a human is within 1 meter, a spatial awareness check for safe robot behavior. Scored by accuracy. Gemini Robotics ER 2 93%
- Moment finding (Google DeepMind) In use Models find the moment in a robot video when an event happens. Scored by accuracy and mean absolute time distance in seconds. Gemini Robotics ER 2 91.3%
- Progress classification (Google DeepMind) In use Models assign each frame of a robot video to one of five task-progress bands from 0 to 100 percent. Scored by accuracy. Gemini Robotics ER 2 57.4%
- LIBERO Fading Simulated Franka tabletop tasks in four 10-task suites. Policies fine-tuned on 50 demos per task. Success rate over ~50 trials. MolmoAct 2 97.2%