Benchmark Almanac

DreamGen Bench

Not in model cards

Video generation benchmark measuring how well video models adapt to a target robot embodiment. Part of NVIDIA's DreamGen pipeline.

Paper reports strong correlation between benchmark score and downstream policy success.

In the authors’ words

We introduce DreamGen, a simple yet highly effective 4-stage pipeline for training robot policies that generalize across behaviors and environments through neural trajectories - synthetic robot data generated from video world models. DreamGen leverages state-of-the-art image-to-video generative models, adapting them to the target robot embodiment to produce photorealistic synthetic videos of familiar or novel tasks in diverse environments. Since these models generate only videos, we recover pseudo-action sequences using either a latent action model or an inverse-dynamics model (IDM). Despite its simplicity, DreamGen unlocks strong behavior and environment generalization: a humanoid robot can perform 22 new behaviors in both seen and unseen environments, while requiring teleoperation data from only a single pick-and-place task in one environment. To evaluate the pipeline systematically, we introduce DreamGen Bench, a video generation benchmark that shows a strong correlation between benchmark performance and downstream policy success. Our work establishes a promising new axis for scaling robot learning well beyond manual data collection. Code available at https://github.com/NVIDIA/GR00T-Dreams.

Abstract of DreamGen: Unlocking Generalization in Robot Learning through Video World Models (arXiv:2505.12705, CC0)
Made by NVIDIA GEAR Released May 2025 Video world models All tasks public
Scored
Graded by a model A language model scores each response, usually against a rubric.
Answers
Open-ended Free-form text, documents, designs or media judged as a whole.

No scores here yet

No lab, leaderboard or Epoch AI run has reported a score for DreamGen Bench yet.

No lab model card reports it yet

We read the headline results of 479 launch posts from 51 labs, and none of them list this benchmark.

More in World models

All 34 in World models
  • AgentWorldBench In use Language world models predict the next environment observation after an agent action, in 7 domains from MCP and search to Android and desktop OS. A judge model scores each prediction against the real observation on five dimensions. Qwen-AgentWorld-35B-A3B 56.4
  • IntPhys 2 In use Models watch simulated videos and judge whether each event is physically possible, testing object permanence, immutability, continuity and solidity. Scored by accuracy. Muse Spark Open 68.3%
  • QwenWorldBench Fading Models act as world models that simulate agent environments in 7 domains, such as terminal, web, and Android. A rubric judge scores five dimensions. Qwen3.7-Plus 62.1%
  • VPCT Fading Models view an image of a ball, ramps, and three buckets and predict which bucket the ball falls into. Scored by accuracy on 100 problems. Gemini 3 Pro 91%
  • ContPhy Fading Models infer physical properties such as mass and density of fluids and soft bodies from videos and predict their dynamics. Seed2.0 Pro 67.4%
  • MVPBench Dropped Models answer video questions on spatial, temporal and intuitive physics. Each question comes in a pair of near-identical videos with opposite answers, so text or single-frame shortcuts fail. gemini-robotics-er-1.5-preview 72.5%