HourVideo
Multimodal models answer questions about first-person videos that run 20 to 120 minutes. The 500 videos come from Ego4D, and the 12,976 five-way multiple-choice questions cover summarization, perception, visual reasoning and navigation. Score is accuracy.
NeurIPS 2024 Datasets and Benchmarks track. Human experts scored 85.0% on a subset, far above every model tested at launch. About 150 GitHub stars.
- Made by
- Stanford University
- Released
- November 2024
- Kind
- Long videos
- Level
- Everyday
- Scored
- Checked automatically
- Answers
- Multiple choice
- Tasks
- All public
In the authors’ words
We present HourVideo, a benchmark dataset for hour-long video-language understanding. Our dataset consists of a novel task suite comprising summarization, perception (recall, tracking), visual reasoning (spatial, temporal, predictive, causal, counterfactual), and navigation (room-to-room, object retrieval) tasks. HourVideo includes 500 manually curated egocentric videos from the Ego4D dataset, spanning durations of 20 to 120 minutes, and features 12,976 high-quality, five-way multiple-choice questions. Benchmarking results reveal that multimodal models, including GPT-4 and LLaVA-NeXT, achieve marginal improvements over random chance. In stark contrast, human experts significantly outperform the state-of-the-art long-context multimodal model, Gemini Pro 1.5 (85.0% vs. 37.3%), highlighting a substantial gap in multimodal capabilities. Our benchmark, evaluation toolkit, prompts, and documentation are available at https://hourvideo.stanford.edu
Gemini 1.5 Pro (snapshot unspecified) tops the leaderboard with 37.3%
Best score per entry from the benchmark's own leaderboard, read on November 12th, 2026. Where a model ran with several settings or harnesses, we keep its best. Names in gray aren't in our model list: fine-tunes, agent systems, products and older models. They don't count toward leaders.
- 1 Gemini 1.5 Pro (snapshot unspecified) 37.3%
- 2 GPT-4 (snapshot unspecified) 25.7%
- 3 LLaVA-34B-DPO 22.3%
Source: Official HourVideo leaderboard
| # | Entry | Score | Setting | Date | Source |
|---|---|---|---|---|---|
| 1 | Gemini 1.5 Pro (snapshot unspecified) | 37.3% | Multimodal Models | Official leaderboard | |
| 2 | GPT-4 (snapshot unspecified) | 25.7% | Socratic Models | Official leaderboard | |
| 3 | LLaVA-34B-DPO | 22.3% | Socratic Models | Official leaderboard |
No lab model card reports it yet
We read the headline results of 522 launch posts from 52 labs, and none of them list this benchmark.
More in Audio and video
- Video-MMMU In use 300 expert-level lecture videos with questions testing perception, comprehension and adaptation of knowledge. Accuracy.
Qwen 3.8 Max 88.7% - LVBench In use Hours-long video understanding with multiple-choice questions on reasoning, grounding and summarization.
Gemini 4 Argon 91.7% - MMVU In use 3,000 expert-annotated questions on videos across 27 disciplines. Multiple-choice and open-ended accuracy. Qwen 3.8 Max 82.4%
- MLVU In use Multi-task long video understanding (minutes to hours) with multiple-choice and generation tasks. Accuracy. Qwen 3.8 Max 90.8%
- FLEURS In use Models transcribe read speech in 102 languages built from FLoRes-101 sentences. Model cards usually report word error rate, so lower scores are better.
Phi-4-multimodal 4.00% - VoiceBench In use Spoken instructions with speaker, environment and content variation. Voice assistant responses scored by GPT judge or rules. Qwen3.5-Omni-Plus 93.1%