Audio and video
We track 168 benchmarks in audio and video, or 175 if you count each version. They run from everyday level to expert level.
Google DeepMind held the top audio and video score 26% of the contested time
That's about 12 of the 46 months since January 2023, averaged over the benchmarks below. The easiest benchmarks are at the top. Versions of one test share a row, with a dark tick at each new version. We stop the clock, and fade the rest of the row, when the best score passes 85%, or at the newest score if 45 days go by without another.
Google DeepMind OpenAI Qwen ByteDance Seed Moonshot AI Shanghai AI Lab Alibaba Group Other labs Z.ai Fixie.ai (Ultravox) Thinking Machines Microsoft Xiaomi Corp Baidu Amazon Anthropic
Everyday Most adults, without training
University Undergraduates and entry-level professionals
Share of contested time at number one
Google DeepMind 26% 
Qwen 21% OpenAI 18% 
ByteDance Seed 17% 
Fixie.ai (Ultravox) 4% Moonshot AI 3% Other labs 3% 
Shanghai AI Lab 3% Anthropic 1% 
Alibaba Group 1% 
Baidu 1% 
Thinking Machines 1% Closed out: 6 benchmarks passed 85%
Google DeepMind 4 Qwen 2 - Aug 2026 LVBench Gemini 3.7 Flash
- Aug 2026 1H-VideoQA Gemini 3.7 Flash
- Feb 2026 MLVU Qwen3.5 397B-A17B
- Nov 2025 Video-MMMU Gemini 3 Pro
- Sep 2025 VoiceBench Qwen3-Omni-30B-A3B
- Jun 2025 Video-MME Gemini 2.5 Pro Jun 2025
Scores come from each benchmark’s own leaderboard, from Epoch AI’s runs and from labs’ reports on their own models. We date Epoch runs by the model’s release and lab reports by when they came out. Labs pick their own settings, so a lab’s claim can beat an independent run of the same model. Turn off lab claims to see independent runs only.
Google DeepMind OpenAI ByteDance Seed Shanghai AI Lab Alibaba Group Other labs Qwen Z.ai Fixie.ai (Ultravox) Moonshot AI NVIDIA Thinking Machines xAI Anthropic Amazon
Everyday Most adults, without training
University Undergraduates and entry-level professionals
Share of contested time at number one, independent runs only
Google DeepMind 55% Fixie.ai (Ultravox) 11% OpenAI 10% Anthropic 4% Alibaba Group 4% xAI 4% ByteDance Seed 3% Other labs 3% Thinking Machines 2% Qwen 2% Shanghai AI Lab 1% Moonshot AI 1% Only each benchmark’s own leaderboard and Epoch AI’s runs, dated by the model’s release. We leave out benchmarks that only labs report.
From everyday to expert level
Grouped by who could solve the tasks, from things most adults can do to open research problems. The bar under each name is its best score.
Everyday
Most adults, without training
- Video-MME
- LVBench
- LMArena Text-to-Video Arena
- LMArena Image-to-Video Arena
- VideoMME v2 (with subtitles)
- LongVideoBench
- MLVU
- TOMATO
- VoiceBench
- MVBench
- EgoSchema
- Audio MC
- 1H-VideoQA
- MotionBench
- OVBench
- TempCompass
- TVBench
- Daily-Omni
- FLEURS
- ActivityNet-QA
- τ-Voice
- AHELM
- Common Voice
- LMArena Video Edit Arena
- Video-Holmes
- WorldSense (omnimodal)
- LibriSpeech
- OVO-Bench
- EgoTempo
- Perception Test MCVQA
- QVHighlights
- VoiceCodeBench
- WenetSpeech
- AISHELL-1
- LiveSports-3K
- StreamingBench
- VATEX
- VATEX ZH
- YouCook2
- AISHELL-2
- CG-Bench
- Charades-STA
- Clotho-AQA
- VideoReasonBench
- ViSpeak-Bench
- WildSpeech-Bench
- AVUT
- CrossVid
- JointAVBench
- Omni-Cloze
- OmniBench
- OpenAudioBench
- QIVD
- AV-SpeakerBench
- MMBench-Video
- MORSE-500
- ODV-Bench
- Seed-TTS-eval
- VideoEval-Pro
- AVMeme Exam
- DREAM-1K
- EgoLife (with memory)
- FutureOmni
- IntelligentBench
- LongAudioSpan
- LSMDC
- LVBench (with memory)
- LVOmniBench
- MAVERIX
- MME-VideoOCR
- MultiChallenge Audio (OpenAI)
- Multilingual LibriSpeech
- Neptune
- NExT-QA
- OmniCap-IF
- OmniMMI
- Open ASR Leaderboard
- SpeechRole
- SpotSoundBench
- StreamBench (StreamChat)
- TACoS
- TemporalBench
- TVQA
- UNO-Bench
- URO-Bench
- VBench
- VCR-Bench
- VideoHallucer
- AIR-Bench (Audio Instruction Benchmark)
- EVA-Bench
- Golden3 speech summarization (Microsoft)
- Kimi-Audio speech conversation evaluation
- MMOU
- MU-Bench
- MultiChallenge Audio (Xiaomi)
- Real-time Audio-Visual Interaction Evaluation
- TimeLens-Bench
- AA-Video-Editing
- APM-Bench
- Artificial Analysis Speech Arena
- Artificial Analysis Video Arena
- Code2Video-Compass
- CodeCap-Bench
- Coval Voice AI Benchmarks
- CoVEBench
- CV3-Eval
- DesignArena Audio Realism Bench
- DesignArena Image to Video
- DesignArena Multi-Image to Video
- DesignArena Multimodal to Video
- DesignArena Speech-to-Speech
- DesignArena Text-to-Speech
- DesignArena Video Arena
- DesignArena Video Editing
- Duplex-MPE
- Full-Duplex-Bench
- Full-Duplex-Bench v1.5
- Full-Duplex-Bench v2
- Full-Duplex-Bench v3
- InstructTTSEval
- KeyFrame-Compass
- Kimi-ASR Internal Testset
- Kontext Bench
- LongAV-Compass
- MAEB
- MeViS
- MiniMax TTS Multilingual Test Set
- MMAE
- Movie Gen Video Bench
- MSEB
- Music Arena
- Omni-InteractScope
- OmniVBench
- OVO-S-Bench
- Q-Bench-Video
- Qwen-Image-Bench
- ReasonVOS
- Ref-DAVIS17
- Ref-YouTube-VOS
- SemComp-Bench
- SoccerNet-Caption
- StreamEval
- TTS Arena
- VBench-2.0
- vCX Benchmark
- Video Reality Test
- Video-Oasis
- VideoFDB
- VoxMem
- VSI-SUPER
School
Students, from grade school to high school
University
Undergraduates and entry-level professionals
Expert
Specialists with years of training, often over hours of work
By kind
Video understanding
| Benchmark | Level | Scored | Best score | Status |
|---|---|---|---|---|
| Video-MME Models answer 2,700 multiple-choice questions about 900 videos from 11 seconds to 1 hour long, with and without subtitles. Scored by accuracy. | Everyday | Checked automatically | 90.4% | In use |
| Video-MMMU 300 expert-level lecture videos with questions testing perception, comprehension and adaptation of knowledge. Accuracy. | University | Checked automatically | 88.7% | In use |
| MMVU 3,000 expert-annotated questions on videos across 27 disciplines. Multiple-choice and open-ended accuracy. | University | Checked automatically | 82.4% | In use |
| VideoMME v2 (with subtitles) Models answer multiple choice questions about videos with subtitles or audio added. Related questions are scored as groups to penalize guessing. | Everyday | Checked automatically | 68.3% | In use |
| TOMATO Models answer 1,484 human-written questions about 1,417 videos that need visual temporal reasoning, such as action counts, direction, rotation and velocity. Scored by accuracy. | Everyday | Checked automatically | 79.5% | Fading |
| MVBench Models answer multiple-choice questions across 20 temporal video tasks that a single frame cannot solve. Scored by accuracy. | Everyday | Checked automatically | 79% | Fading |
| MotionBench Models answer questions about fine-grained motion in video clips across six motion question types. Scoring is accuracy. | Everyday | Checked automatically | 75.2% | Fading |
| OVBench Video models answer questions about past, current and future moments of an online video stream in 6 task types and 16 subtasks. | Everyday | Checked automatically | 70% | Fading |
| TempCompass Video models answer questions about temporal aspects of short clips, such as action order, speed, direction and attribute change, in several question formats. | Everyday | Checked automatically | 89.6% | Fading |
| TVBench Models answer multiple-choice questions on 10 temporal video tasks, such as action counting and ordering, that static frames cannot solve. Scored by accuracy. | Everyday | Checked automatically | 81.9% | In use |
| Daily-Omni Models answer 1,197 multiple choice questions about everyday videos that need audio and visual events aligned in time. Scored by accuracy. | Everyday | Checked automatically | 85.1% | Fading |
| ActivityNet-QA Models answer open-ended questions about web videos of human activities, from 58,000 QA pairs on 5,800 videos. Scored by accuracy. | Everyday | Graded by a model | 66.7% | Dropped |
| Video-Holmes Models answer 1,837 questions about 270 suspense short films across seven reasoning tasks. Each question needs several visual clues from different parts of the film. | Everyday | Checked automatically | 68.2% | Fading |
| WorldSense (omnimodal) Models answer multiple choice questions about real-world videos that need both the visual and the audio track. Scored by accuracy. | Everyday | Checked automatically | 68.5% | Fading |
| OVO-Bench Video models answer questions asked at given timestamps of a video stream in 12 tasks over 644 videos. Tasks cover backward tracing, real-time understanding and forward active responding. | Everyday | Checked automatically | 80.7% | Fading |
| EgoTempo Models answer questions about egocentric videos that require temporal understanding across the whole clip. | Everyday | Checked automatically | 71.8% | Fading |
| Perception Test MCVQA Models answer multiple-choice questions about real-world videos filmed to show perceptually interesting situations. Scored by top-1 accuracy. | Everyday | Checked automatically | 78.4% | Dropped |
| QVHighlights Natural-language queries over YouTube videos. Model localizes relevant moments, scored by Recall@1 at IoU 0.5. | Everyday | Checked automatically |  77.2% | Fading |
| VideoDR (with search) Agents watch a video, gather clues across frames, and search the web to answer multi-hop questions. Scored by answer accuracy. | University | Checked automatically | 76% | In use |
| StreamingBench Models answer questions while a video streams in, as a human viewer would. It tests real-time visual understanding, omni-source understanding and context understanding. | Everyday | Checked automatically | 84.4% | Fading |
| Charades-STA Models read a sentence describing an action and return its start and end time in a Charades indoor video. Scored by temporal IoU with the labeled moment. | Everyday | Checked automatically | 64% | Dropped |
| VideoReasonBench Video models watch a sequence of fine-grained operations on a partly hidden state and answer questions that recall, infer or predict that state. | Everyday | Checked automatically | 77.8% | Fading |
| ViSpeak-Bench Streaming video models must notice instructions given visually, such as a hand wave, and respond in seven subtasks of visual instruction feedback. | Everyday | Checked automatically | 87% | Fading |
| AVUT Models answer questions about videos where the audio carries the key information. The benchmark removes questions that text alone can answer. | Everyday | Checked automatically | 86.6% | Fading |
| CrossVid Models reason across several videos at once in 10 tasks, with 9,015 questions on 5,331 videos in single-choice, multiple-choice and open-ended formats. | Everyday | Checked automatically | 65% | Fading |
| JointAVBench Models answer questions about videos that cannot be answered from vision or audio alone. Questions cover speech and sound events over several scene spans. | Everyday | Checked automatically | 75.9% | Fading |
| OmniVideoBench Models answer 1,000 questions about videos that need audio and visual evidence together. Each question has annotated reasoning steps. | University | Checked automatically | 63.4% | Fading |
| QIVD Models answer spoken questions about short phone videos in real time, as in a face to face talk. Each item has a short answer and a timestamp. | Everyday | Graded by a model | 69.6% | Fading |
| AV-SpeakerBench Models answer 3,212 multiple-choice questions about who speaks, what is said and when in real-world videos. Questions need audio and video together. | Everyday | Checked automatically | 77.2% | Fading |
| MORSE-500 Models answer questions embedded in 500 scripted video clips across six reasoning categories, including abstract, physical, planning, spatial and temporal reasoning. | Everyday | Checked automatically | 37.4% | Fading |
| ODV-Bench Streaming video models answer real-time questions about autonomous driving footage. | Everyday | Checked automatically | 72.5% | Fading |
| FutureOmni Models watch and hear 919 videos and answer 1,034 multiple-choice questions that ask what will happen next. Questions need audio and visual cues together. | Everyday | Checked automatically | 66.7% | Fading |
| LSMDC Models fill in a missing word in movie audio descriptions using the matching video clip. Scoring is accuracy. | Everyday | Checked automatically | 45.7% | Dropped |
| MAVERIX Models answer 2,556 questions about 700 videos that need both the video and its audio. Questions come in multiple-choice and open-ended forms. | Everyday | Checked automatically | 66.6% | Fading |
| MME-VideoOCR Models answer 2,000 questions about text that appears in 1,464 videos, across 25 tasks from recognition to reasoning. | Everyday | Checked automatically | 77% | Fading |
| NExT-QA Models answer causal, temporal and descriptive questions about short videos of daily life. Labs usually report the multiple choice split. | Everyday | Checked automatically |  86.2% | Dropped |
| OmniMMI Omni models answer 2,290 questions on 1,121 streaming videos, including proactive tasks where they must decide when to respond. | Everyday | Checked automatically | 53% | Fading |
| StreamBench (StreamChat) Video models answer questions on streaming videos across media types, including multi-turn interaction and reasoning tasks. | Everyday | Checked automatically | 72.8% | Dropped |
| TACoS Models find the time span in a cooking video that matches a sentence. Labs score temporal grounding by mean IoU with the annotated span. | Everyday | Checked automatically | 49.6% | Dropped |
| TemporalBench Video models answer about 10K questions on fine-grained temporal dynamics, built from about 2K detailed human annotations of video clips. | Everyday | Checked automatically | 79.8% | Dropped |
| TVQA Models answer 152,545 multiple-choice questions about clips from 6 TV shows, using video frames and subtitles. Scoring is accuracy. | Everyday | Checked automatically | 87.3% | Dropped |
| VCR-Bench Video models answer 1,034 questions on 859 videos, and graders score both the final answer and each tagged chain-of-thought step against human rationales. | Everyday | Graded by a model | 59.8% | Fading |
| VideoHallucer Paired basic and hallucinated questions about videos detect intrinsic and extrinsic hallucinations. Scored by pair accuracy. | Everyday | Checked automatically | 59.6% | Fading |
| TimeLens-Bench Models find the start and end time of a described event in a video. The benchmark re-annotates Charades-STA, ActivityNet Captions and QVHighlights with strict quality rules, and scores by mean IoU. | Everyday | Checked automatically | Fading | |
| BVB Multimodal agents rebuild real-world videos as animated Blender scenes by writing code. The renders are compared with the source video under a shared cost limit. | Expert | Checked automatically | Not in cards | |
| CineRecon Agents use tools to rebuild reference videos from film, real footage and games as editable dynamic 3D scenes. The set has 50 videos and 360 evaluation clips. | Expert | Graded by a model | Not in cards | |
| MeViS Models segment the objects in a video that a sentence picks out by their motion, not by static looks. Scored by J&F. | Everyday | Checked automatically | Not in cards | |
| Minerva-Ego Models answer 1,160 multiple-choice questions on egocentric kitchen videos from HD-EPIC. Each question has a hand-written reasoning trace grounded in time and objects. | Expert | Checked automatically | Not in cards | |
| Omni-InteractScope Omnimodal models watch continuous video and audio streams and must respond at the right moment with answers, alerts, warnings, tool calls or step guidance. Scoring joins content and timing. | Everyday | Graded by a model | Not in cards | |
| OVO-S-Bench Multimodal models answer spatial questions about places and layouts during a continuous egocentric video stream. The set has 1,680 human-annotated questions over 348 videos. | Everyday | Checked automatically | Not in cards | |
| Q-Bench-Video Models judge the quality of natural, AI-generated and computer-graphics videos through yes-or-no, what-how, open-ended and pairwise comparison questions. | Everyday | Graded by a model | Not in cards | |
| ReasonVOS Models segment the object in a video that an implicit query describes, where finding it needs world knowledge and reasoning. Scored by J&F. | Everyday | Checked automatically | Not in cards | |
| Ref-DAVIS17 Models segment and track the object that a sentence describes through each frame of a DAVIS 2017 video. Scored by J&F, the mean of mask overlap and contour accuracy. | Everyday | Checked automatically | Not in cards | |
| Ref-YouTube-VOS Models segment and track the object that a sentence describes in YouTube-VOS videos. Scored by J&F on a server-graded validation split. | Everyday | Checked automatically | Not in cards | |
| Video Reality Test Vision-language models and humans judge whether ASMR videos with tight audio and visual coupling are real or AI-generated. It tests perceptual realism detection. | Everyday | Checked automatically | Not in cards | |
| Video-Oasis A diagnostic suite that audits video understanding benchmarks and keeps only samples that need visual and temporal input. Models are tested on the remaining video-native questions. | Everyday | Checked automatically | Not in cards |
Long videos
| Benchmark | Level | Scored | Best score | Status |
|---|---|---|---|---|
| LVBench Hours-long video understanding with multiple-choice questions on reasoning, grounding and summarization. | Everyday | Checked automatically | 91.7% | In use |
| LongVideoBench Models answer referring reasoning questions over 3,763 web videos up to an hour long, with interleaved subtitles. Scored by multiple-choice accuracy. | Everyday | Checked automatically | 80.6% | Fading |
| MLVU Multi-task long video understanding (minutes to hours) with multiple-choice and generation tasks. Accuracy. | Everyday | Checked automatically | 90.8% | In use |
| EgoSchema Models answer five-way multiple-choice questions about three-minute egocentric video clips from Ego4D. Scored by accuracy on 5,031 questions. | Everyday | Checked automatically | 76.2% | Dropped |
| 1H-VideoQA Models answer questions about videos about one hour long, so the answer needs evidence from far parts of the video. | Everyday | Checked automatically | 89.1% | System cards only |
| MINERVA Models answer complex video questions that need multi-step reasoning, with reference reasoning traces for grading. | University | Graded by a model | 70.7% | Fading |
| CG-Bench Video models answer 12,129 questions about 1,219 long videos and must also find the clue segments that support each answer. | Everyday | Checked automatically | 65.5% | Fading |
| MMBench-Video Video models answer free-form questions about long, multi-shot YouTube videos. A GPT-4 judge scores the answers. | Everyday | Graded by a model | 2.02 | Dropped |
| VideoEval-Pro Video models give open-ended short answers to questions that need the whole long video, which removes the guessing that multiple choice allows. | Everyday | Graded by a model | 49.5% | Fading |
| EgoLife (with memory) Models answer life-assistant questions about a week of egocentric video from six people, such as recalling past events and habits. | Everyday | Checked automatically | 80.3% | In use |
| LVBench (with memory) Models answer multiple-choice questions about videos up to two hours long, aided by Qwen's external video memory system. Scored by accuracy. | Everyday | Checked automatically | 85.6% | In use |
| LVOmniBench Models answer questions about long videos with rich audio, often tens of minutes long. The questions need both the audio and the video track. | Everyday | Checked automatically | 63.3% | In use |
| Neptune Models answer multiple-choice questions about long videos that need reasoning over many segments. | Everyday | Checked automatically | 87.3% | System cards only |
| ZeroVideo Models answer real-world video reasoning questions that need high frame rate motion perception or multi-hop reasoning over long videos. Videos average 1,672 seconds. | Expert | Checked automatically | 14.5% | System cards only |
| MMOU Models answer 20,000 questions about 11,877 long web videos, using sound, images and text together. Questions cover 13 skill categories. | Everyday | Checked automatically | 78.8% | System cards only |
| APM-Bench Streaming video assistants must remember experiences across separate sessions of a person's day. The set has 549 sessions in 104 life trajectories with objective and open questions. | Everyday | Graded by a model | Not in cards | |
| MINERVA-Cultural Models answer about 2,200 open-ended questions on 540 videos from 18 locales, written in native languages by local experts. An LLM judge scores each answer 0, 1 or 2. | Expert | Graded by a model | Not in cards | |
| VSI-SUPER Video models watch arbitrarily long streams and must recall where objects appeared (VSR) or keep counting objects across scenes (VSC). Video lengths run from 10 minutes to 4 hours. | Everyday | Checked automatically | Not in cards |
Video captioning
| Benchmark | Level | Scored | Best score | Status |
|---|---|---|---|---|
| LiveSports-3K Video models give real-time commentary and answer questions on sports clips. An LLM judge scores the free-form commentary. | Everyday | Graded by a model | 78.1% | Fading |
| VATEX Models write English or Chinese captions for short video clips from a set of over 41,000 videos. Scored by CIDEr. | Everyday | Checked automatically | 71.3 | Dropped |
| VATEX ZH Models write Chinese captions for short videos of human activities from the VATEX dataset. Captions are scored with CIDEr against human references. | Everyday | Checked automatically | 59.7 | Dropped |
| YouCook2 Models describe the procedure steps in YouTube cooking videos. Scored by CIDEr against human step captions. | Everyday | Checked automatically | 188.3 | Dropped |
| Omni-Cloze Models write detailed captions of audio, visual and audio-visual clips. A cloze test on the caption content scores the detail. | Everyday | Checked automatically | 64.8% | Fading |
| DREAM-1K Video models write fine-grained descriptions of 1,000 videos from varied sources. An automatic method scores how many described events match the reference. | Everyday | Graded by a model | 43.9% | Dropped |
| OmniCap-IF Models write captions for videos with audio while following detailed user constraints. Captions are scored on format and content compliance. | Everyday | Graded by a model | 80.6% | In use |
| CodeCap-Bench Models watch a structured 2D video and write code that re-renders it. The rendered video is compared with the original on visual similarity, object timing and placement, and retained detail. | Everyday | Graded by a model | Not in cards | |
| SoccerNet-Caption Models write timestamped commentary for soccer broadcasts. The set has almost 37k commentaries over 715.9 hours of video. Streaming reports also score when the model chooses to speak. | Everyday | Checked automatically | Not in cards |
Speech recognition and translation
| Benchmark | Level | Scored | Best score | Status |
|---|---|---|---|---|
| CoVoST2 Models translate speech from 21 languages into English and from English into 15 languages. Scored by BLEU. | University | Checked automatically | 40.7 | Fading |
| FLEURS Models transcribe read speech in 102 languages built from FLoRes-101 sentences. Model cards usually report word error rate, so lower scores are better. | Everyday | Checked automatically |  4.00% | Fading |
| τ-Voice Full-duplex voice agents handle real-time support calls in retail, airline and telecom with tools and a simulated caller. Scoring covers task completion and natural interaction. | Everyday | Checked automatically | 67.3% | In use |
| Common Voice Crowd-sourced multilingual read-speech corpus. Used here as an ASR test set scored by word error rate. | Everyday | Checked automatically | 6.80% | Fading |
| LibriSpeech Models transcribe English audiobook speech. Scored by word error rate on the test-clean and test-other splits. | Everyday | Checked automatically |  1.53% | Fading |
| MMSU Models answer questions about spoken language that need its content, its paralinguistic cues such as emotion and pitch, and its phonology. | University | Checked automatically | 86.5% | Fading |
| VoiceCodeBench Speech-to-text models transcribe 300 English workplace recordings. They are scored on whether every target entity, such as an address or email, survives exactly. | Everyday | Checked automatically | 59.7% | Not in cards |
| WenetSpeech Models transcribe Mandarin speech from many domains. Scored by character error rate on the test-net and test-meeting sets. | Everyday | Checked automatically | Fading | |
| AISHELL-1 Models transcribe Mandarin read speech. Scored by character error rate. | Everyday | Checked automatically | 0.31% | Dropped |
| AISHELL-2 Models transcribe 1,000 hours of Mandarin speech recorded on several devices. Scored by character error rate. | Everyday | Checked automatically | 2.56% | Dropped |
| WildSpeech-Bench Models hold spoken conversations built from real chat data, with prosody, homophones and stuttering. A judge model scores the replies. | Everyday | Graded by a model | 75.4 | Fading |
| Seed-TTS-eval Models clone a voice from a short prompt and speak new text in English and Mandarin. Scored by word error rate from an ASR model and by speaker similarity. | Everyday | Checked automatically | Fading | |
| Multilingual LibriSpeech Models transcribe read audiobook speech in eight languages from LibriVox. Scored by word error rate. | Everyday | Checked automatically | 4.70 | Fading |
| Open ASR Leaderboard Speech recognition models transcribe the test sets of several English and multilingual corpora. Scored by average word error rate, with speed reported next to it. | Everyday | Checked automatically | 6.14% | Fading |
| SpeechRole Speech role-playing agents talk in character across 98 roles. A model judge scores interaction ability, speech expressiveness and role fidelity from the generated speech. | Everyday | Graded by a model | 123.5 | Fading |
| URO-Bench Spoken dialogue models answer speech with speech across Understanding, Reasoning and Oral conversation. It has a basic track of 16 datasets and a pro track of 20. | Everyday | Graded by a model | 84.1% | Fading |
| EVA-Bench Voice agents hold simulated spoken conversations to complete enterprise tasks. Scores cover task success and voice-specific failure modes. | Everyday | Graded by a model | In use | |
| Kimi-Audio speech conversation evaluation People rate spoken replies on speed, accent and emotion control, empathy and style control. Scored on a 1 to 5 scale and averaged. | Everyday | Graded by people | 3.90 | System cards only |
| MU-Bench Speech-to-text providers transcribe 4,270 caller utterances from real 8 kHz customer service phone calls in five locales, scored against human word-level transcripts. | Everyday | Checked automatically | 27.3% | Not in cards |
| MultiChallenge Audio (Xiaomi) Models answer the final turn of MultiChallenge conversations converted to speech, with speech-to-text and speech-to-speech versions. Xiaomi built the conversion in house. | Everyday | Graded by a model | 15.2% | System cards only |
| Real-time Audio-Visual Interaction Evaluation Trained conversationalists hold three minute live video calls with a model. Real users rate each session for naturalness and fluency on a 0 to 3 scale. | Everyday | Graded by people | 1.37 | System cards only |
| Coval Voice AI Benchmarks Speech-to-text, text-to-speech and speech-to-speech models from almost 40 providers are scored every 30 minutes on call-like audio with accents, noise and echo. | Everyday | Checked automatically | Not in cards | |
| CV3-Eval Text-to-speech models clone voices from in-the-wild reference audio across languages, dialects and emotions. Scored by ASR error rate, speaker similarity and DNSMOS audio quality. | Everyday | Checked automatically | System cards only | |
| Duplex-MPE Full-duplex speech models join conversations of three or four people and must decide when to answer, stay silent or stop speaking. The set has 2,000 scenarios. | Everyday | Checked automatically | Not in cards | |
| Full-Duplex-Bench Full-duplex spoken dialogue models are scored on pause handling, backchanneling, smooth turn-taking and user interruption. Automatic metrics include takeover rate and latency. | Everyday | Checked automatically | Not in cards | |
| Full-Duplex-Bench v1.5 Full-duplex speech models are tested on overlap scenarios: listener backchannel, side conversation and ambient speech. Metrics cover how the model yields, continues or responds. | Everyday | Checked automatically | Not in cards | |
| Full-Duplex-Bench v2 An automated examiner holds real-time multi-turn audio conversations with the model over WebRTC or WebSocket. A model judge scores task completion under conversational constraints. | Everyday | Graded by a model | Not in cards | |
| Full-Duplex-Bench v3 Voice agents hear real human speech with five kinds of disfluency and must make chained API calls across four task domains. Scored on Pass@1, response quality, latency and turn-taking. | Everyday | Checked automatically | Not in cards | |
| InstructTTSEval Text-to-speech models follow natural-language style instructions in English and Chinese across acoustic parameter, descriptive style and role-play tasks. Gemini judges each clip. | Everyday | Graded by a model | System cards only | |
| Kimi-ASR Internal Testset Models transcribe speech from two internal Moonshot test subsets. Scored by word error rate, lower is better. | Everyday | Checked automatically | System cards only | |
| MiniMax TTS Multilingual Test Set Text-to-speech models clone two Common Voice speakers per language and read 100 test sentences in each of 24 languages. Scored by word error rate and speaker similarity. | Everyday | Checked automatically | System cards only | |
| Qwen in-house voice-chat benchmark Models answer spoken versions of text benchmarks such as MMLU, CEval, IFEval, GSM8K, Math23K and Math401. Each part keeps the source benchmark's scoring. | School | Checked automatically | System cards only | |
| vCX Benchmark Models run inside a Retell voice agent on pass or fail checks for business rules, grounding, tool calls and speech-ready replies. Each suite reports its own pass rate. | Everyday | Checked automatically | Not in cards | |
| VideoFDB Multimodal agents are evaluated on two-person video conference recordings. They cover 11 conversational dynamics across verbal, nonverbal and mixed behavior, annotated by people in three passes. | Everyday | Graded by people | Not in cards |
Audio understanding and voice
| Benchmark | Level | Scored | Best score | Status |
|---|---|---|---|---|
| VoiceBench Spoken instructions with speaker, environment and content variation. Voice assistant responses scored by GPT judge or rules. | Everyday | Graded by a model | 93.1% | In use |
| MMAU 10K audio clips with questions on speech, sound and music needing expert knowledge and reasoning. Multiple-choice accuracy. | University | Checked automatically | 82.2% | In use |
| Audio MC Spoken multi-turn conversations test inference memory, instruction retention, self-coherence and voice editing. O4-mini judge applies per-conversation rubrics. | Everyday | Graded by a model | 60.4% | In use |
| Big Bench Audio Models answer 1,000 spoken questions from four Big Bench Hard tasks: formal fallacies, navigate, object counting and web of lies. Built to test reasoning over audio input. | School | Checked automatically | 98.5% | In use |
| AHELM Audio-language models are scored across 10 aspects with standard prompts. The suite combines existing datasets with two new ones, PARADE for stereotypes and CoRe-Bench for conversational reasoning. | Everyday | Checked automatically | 80.3 | Not in cards |
| MMAU v05.15.25 Models answer multiple-choice questions about speech, sound and music clips that need expert knowledge and reasoning. Scored by accuracy on test-mini or the hidden-answer test set. | University | Checked automatically | 77.8% | Not in cards |
| MuChoMusic Models answer multiple-choice questions about music recordings. It tests music knowledge and reasoning in audio-language models. | University | Checked automatically | 72.6% | Fading |
| Clotho-AQA Models answer crowdsourced yes or no and single word questions about short environmental audio clips. Scored by accuracy. | Everyday | Checked automatically | 73.7% | Dropped |
| MMAR Models answer 1,000 questions about real-world audio of speech, sound, music and their mixes. Questions span four reasoning layers from signal to cultural knowledge. | University | Checked automatically | 80% | Fading |
| MMAU-Pro Models answer 5,305 expert-written questions about speech, sounds, music and their mixes, covering 49 listening skills. Some clips run up to 10 minutes. | University | Checked automatically | 69.4% | Fading |
| OmniBench Models answer multiple choice questions that need an image, an audio clip and text together. Scored by accuracy. | Everyday | Checked automatically | 61.4% | Dropped |
| OpenAudioBench Models answer spoken questions in five sets: Reasoning QA, Llama Questions, Web Questions, TriviaQA and AlpacaEval. Knowledge sets use accuracy and the open sets use a model score. | Everyday | Graded by a model | 80.1% | Dropped |
| Audio Understanding Benchmark (Mistral) Models answer hard questions about 40 long-form audio recordings. Mistral built it in-house to test audio understanding beyond transcription. | Expert | Graded by a model | 86.6% | Dropped |
| AVMeme Exam Models answer questions about over one thousand iconic internet sounds and videos, from surface content to context, emotion, usage and world knowledge. | Everyday | Checked automatically | 69.5% | Fading |
| HumMusQA Models answer 320 hand-written questions about music, curated and checked by people with musical training. | Expert | Checked automatically | 75.8% | In use |
| LongAudioSpan Models answer 3,240 questions about 360 English and Chinese recordings from 10 minutes to over 2 hours. Each question is scored as multiple choice and as open-ended. | Everyday | Graded by a model | 82.7% | In use |
| MultiChallenge Audio (OpenAI) Speech-to-speech models answer an audio-friendly subset of MultiChallenge multi-turn questions that OpenAI converted from text to speech. Scored on instruction following accuracy. | Everyday | Graded by a model | 30.5% | Dropped |
| MusTBench Models answer five kinds of music questions that need the correct time region of the audio, such as instrument entries. Music experts validated the items. | Expert | Checked automatically | 50.6% | In use |
| SpotSoundBench Models find the time span of short sound events hidden in long, unstructured YouTube recordings. It has 400 audio, query and timestamp triplets. | Everyday | Checked automatically | 67.2% | In use |
| UNO-Bench Models answer 1,250 human-written omni-modal questions and 2,480 uni-modal questions across 44 task types and 5 modality mixes. Multiple choice and multi-step open questions are scored, the open ones by a scoring model. | Everyday | Graded by a model | 49.9% | Dropped |
| AIR-Bench (Audio Instruction Benchmark) Audio language models answer questions about speech, natural sounds and music. A foundation set has 19 tasks with about 19k single-choice questions, and a chat set has 2k open-ended questions scored by GPT-4. | Everyday | Graded by a model | 6.98 | System cards only |
| Golden3 speech summarization (Microsoft) Models summarize 108 recorded meetings of about 6 minutes each from 1071 instructions. GPT-4 rates overall quality from 1 to 7. | Everyday | Graded by a model | 6.28 | System cards only |
| Artificial Analysis Speech Arena Voters listen to two speech clips made from the same text and pick the more natural one. Votes become Elo ratings. | Everyday | Ranked by votes | Not in cards | |
| DesignArena Audio Realism Bench Voice models produce conversational speech, and users vote on which sounds more real. DesignArena built it because 1 to 5 opinion scores for TTS had saturated. | Everyday | Ranked by votes | Not in cards | |
| DesignArena Speech-to-Speech Voice models answer spoken input with speech. Users listen to two anonymous replies and vote for the better one. | Everyday | Ranked by votes | Not in cards | |
| DesignArena Text-to-Speech Speech models read the same text aloud. Users listen to two anonymous clips and vote for the more natural one. | Everyday | Ranked by votes | Not in cards | |
| MAEB Audio embedding models are scored on 30 tasks across speech, music, environmental sound and audio-text retrieval in over 100 languages. | Everyday | Checked automatically | Not in cards | |
| MMAE Audio models edit audio from instructions across a broad range of real-world editing tasks and sound types. | Everyday | Graded by a model | Not in cards | |
| MSEB Audio representations from any multimodal system are scored on eight core tasks, including transcription, classification, retrieval, reasoning, segmentation, clustering, reranking and reconstruction. | Everyday | Checked automatically | Not in cards | |
| Music Arena Users write a music prompt, listen to tracks from two text-to-music systems, and vote. Preferences, listening data and feedback feed a leaderboard. | Everyday | Ranked by votes | Not in cards | |
| TTS Arena Users listen to two anonymous text-to-speech models read the same text and vote for the more natural one. Votes become Elo ratings. | Everyday | Ranked by votes | Not in cards | |
| VoxMem Audio language models answer questions that need memory of earlier spoken sessions. Questions cover what was said, who said it, how it was spoken and what was audible. | Everyday | Checked automatically | Not in cards |
Video generation
| Benchmark | Level | Scored | Best score | Status |
|---|---|---|---|---|
| LMArena Text-to-Video Arena Video generation models make clips from the same text prompt, anonymously and side by side. Votes from logged-in users give Elo-style ratings. | Everyday | Ranked by votes | 1,516 | Not in cards |
| LMArena Image-to-Video Arena Video generation models animate the same input image with the same prompt, anonymously and side by side. Votes give Elo-style ratings. | Everyday | Ranked by votes |  1,495 | Not in cards |
| Code2Video Benchmark 168 product-video briefs. Agent writes HTML motion-graphics code rendered by HyperFrames. A trained pairwise judge on five axes produces Elo. | Expert | Graded by a model | 1,585 | Not in cards |
| LMArena Video Edit Arena Video editing models apply the same edit request to the same clip, anonymously and side by side. Community votes give Elo-style ratings. | Everyday | Ranked by votes | 1,414 | Not in cards |
| VibeVideoBench Agents write code to render a finished video with sound, with no image or video generation model, in their native harness within one hour. Program checks, task rubrics and a judge panel score the result. | Expert | Graded by a model | 4.11 | Not in cards |
| AgenticVBench Agents complete 100 video post-production tasks in 4 task families built from real workflows by 20 industry experts. Programmatic verifiers and expert rubrics score the results. | Expert | Graded by a model | 36.8% | In use |
| IntelligentBench Image editing models complete 350 free-form edits that need reasoning and world knowledge. GPT-4o scores each output from 0 to 2 against a reference image, normalized to 100. | Everyday | Graded by a model | 55.3 | Dropped |
| VBench Video generation models make clips from prompts, and automatic checks score 16 dimensions of quality and prompt adherence. | Everyday | Checked automatically | 84.2% | Fading |
| AA-Video-Editing Voters compare two edits of the same video made from one instruction and pick the better one. Votes become Elo ratings. | Everyday | Ranked by votes | Not in cards | |
| Artificial Analysis Video Arena Voters compare videos from two text-to-video models for the same prompt and pick the better one. Votes become Elo ratings. | Everyday | Ranked by votes | Not in cards | |
| Code2Video-Compass Multimodal models write programs that render complete videos from text, an image or a video. | Everyday | Graded by a model | Not in cards | |
| CoVEBench Video editing models follow compositional instructions with several coupled edits to subjects, actions and camera, while unrelated content must stay unchanged. | Everyday | Graded by a model | Not in cards | |
| DesignArena Image to Video Video models animate a user image into a short video. Users compare two anonymous videos and vote for the better one. | Everyday | Ranked by votes | Not in cards | |
| DesignArena Multi-Image to Video Video models compose one video from several reference images. Users compare two anonymous videos and vote for the better one. | Everyday | Ranked by votes | Not in cards | |
| DesignArena Multimodal to Video Video models make a video from a mix of text, images and other inputs. Users compare two anonymous videos and vote for the better one. | Everyday | Ranked by votes | Not in cards | |
| DesignArena Video Arena Users compare videos from anonymous video generation models for the same prompt and vote for the better one. Votes become Elo ratings. | Everyday | Ranked by votes | Not in cards | |
| DesignArena Video Editing Video models edit a user video following a text instruction. Users compare the two results and vote for the better one. | Everyday | Ranked by votes | Not in cards | |
| KeyFrame-Compass Video generators must reproduce a given sequence of keyframes while keeping overall video quality. The set has 386 samples over three domains and four keyframe densities. | Everyday | Graded by a model | Not in cards | |
| Kontext Bench Image editing models get 1,026 image and instruction pairs across local edits, global edits, text editing, style reference and character consistency. | Everyday | Graded by people | Not in cards | |
| LongAV-Compass Models generate minute-long videos with sound from text, an image or a video. Scoring covers identity consistency, narrative coherence and audio-visual alignment over time. | Everyday | Graded by a model | Not in cards | |
| Movie Gen Video Bench Video generation models make clips from a fixed prompt set, and people compare them. Meta released non-cherry-picked Movie Gen videos for comparison. | Everyday | Graded by people | Not in cards | |
| OmniVBench Video generators make videos that follow many kinds of reference inputs at once. Scoring checks that each reference factor is kept, separated and placed correctly. | Everyday | Graded by a model | Not in cards | |
| Qwen-Image-Bench Text-to-image models are tested on professional creative work, from faithful real-world reconstruction to creative expression. A released judge model scores the images. | Everyday | Graded by a model | Not in cards | |
| SemComp-Bench Video generation models must complete an intended task outcome from a reference image while keeping the high-level semantics of the reference. | Everyday | Graded by a model | Not in cards | |
| StreamEval Streaming video generation models handle 300 scripted cases of 15 to 300 seconds. Requests arrive over time across avatar dialogue, performance, world generation and editing. | Everyday | Graded by a model | Not in cards | |
| VBench-2.0 Video generation models are scored on intrinsic faithfulness, such as human fidelity, controllability, creativity, physics and commonsense, beyond pixel quality and basic prompt match. | Everyday | Graded by a model | Not in cards |