AV-SpeakerBench

Qwen3.8-Omni-Flash 77.2% leads since Sep 2026 Fading

Models answer 3,212 multiple-choice questions about who speaks, what is said and when in real-world videos. Questions need audio and video together.

Qwen reports it in the Qwen3.5-Omni audio-visual table.

In the authors’ words

Multimodal large language models (MLLMs) are expected to jointly interpret vision, audio, and language, yet existing video benchmarks rarely assess fine-grained reasoning about human speech. Many tasks remain visually solvable or only coarsely evaluate speech, offering limited insight into whether models can align who speaks, what is said, and when it occurs. We introduce AV-SpeakerBench, a curated benchmark of 3,212 multiple-choice questions focused on speaker-centric audiovisual reasoning in real-world videos. It features: (1) a speaker-centered formulation that treats speakers-not scenes-as the core reasoning unit; (2) fusion-grounded question design embedding audiovisual dependencies into question semantics; and (3) expert-curated annotations ensuring temporal precision and cross-modal validity. Comprehensive evaluations show that the Gemini family consistently outperforms open-source systems, with Gemini 2.5 Pro achieving the best results. Among open models, Qwen3-Omni-30B approaches Gemini 2.0 Flash but remains far behind Gemini 2.5 Pro, primarily due to weaker audiovisual fusion rather than visual perception. We believe AV-SpeakerBench establishes a rigorous foundation for advancing fine-grained audiovisual reasoning in future multimodal systems.

Abstract of See, Hear, and Understand: Benchmarking Audiovisual Human Speech Understanding in Multimodal Large Language Models (arXiv:2512.02231, CC0)

Made by Nguyen et al. (arXiv 2512.02231) Released Dec 2025 Video understanding All tasks public Last in a launch post Mar 2026 , dropped before it saturated Last model added Sep 2026

Level: Everyday Most adults, without training
Scored: Checked automatically A program checks the result against an answer key, unit tests or the expected final state.
Answers: Multiple choice

plnguyen2908/AV-SpeakerBench 12 stars Paper

Models scored in lab cards

Launch post lists it, from 1 lab

System cards report it, from 1 lab

Qwen3.8-Omni-Flash leads with 77.2%

Best score per model as labs report them in their own cards. Labs pick their own settings, so these compare less cleanly than independent runs.

Bars Table

1 Qwen3.8-Omni-Flash 77.2%
2 Qwen3.5-Omni-Plus 71.3%

#	Model	Score	Setting	Date	Source
1	Qwen3.8-Omni-Flash	77.2%		Sep 2026	Model card
2	Qwen3.5-Omni-Plus	71.3%		Mar 2026	Model card

2 models have held number one since its release in Dec 2025

Lab claims count from the day the card came out, independent runs from the model’s release.

Qwen

Lab claims 2 leaders

Qwen3.8-Omni-Flash 77.2%

Scores over time

All the scores we have, one dot each, so a model tested at several settings appears more than once.

Reported by the lab in its model card Best so far

Labs headline it less often now, last in Mar 2026

Launch posts from every lab that has listed it, one tick per post. Tall colored ticks include it, and short gray ones leave it out.

Qwen

Date	Launch post	Score	As the post writes it
30 Mar 2026	Qwen3.5-Omni-Plus	71.3	AV-SpeakerBench

2 system cards report it, from 1 lab

Besides launch posts, we read the system cards, model cards and technical reports labs publish with a release. They report far more, including safety and frontier-risk tests.

Date	Document	Score	As the document writes it	Section
22 Sep 2026	Qwen3.8-Omni-Flash technical report	77.2	AVSpeaker, cited as arXiv 2512.02231 (Table 5)	Capabilities
17 Apr 2026	Qwen3.5-Omni-Plus technical report	71.3	AV-SpeakerBench (Table 7)	Capabilities

Qwen3.8-Omni-Flash leads with 77.2%

2 models have held number one since its release in Dec 2025

Scores over time

Labs headline it less often now, last in Mar 2026

2 system cards report it, from 1 lab

More in Audio and video