Benchmark Almanac

Omni-Cloze

Qwen3.5-Omni-Plus 64.8% leads since Mar 2026 Fading

Models write detailed captions of audio, visual and audio-visual clips. A cloze test on the caption content scores the detail.

Released with the Omni-Captioner paper. Qwen3.5-Omni and Qwen3.8-Omni-Flash report it.

In the authors’ words

Fine-grained perception of multimodal information is critical for advancing human-AI interaction. With recent progress in audio-visual technologies, Omni Language Models (OLMs), capable of processing audio and video signals in parallel, have emerged as a promising paradigm for achieving richer understanding and reasoning. However, their capacity to capture and describe fine-grained details remains limited explored. In this work, we present a systematic and comprehensive investigation of omni detailed perception from the perspectives of the data pipeline, models, and benchmark. We first identify an inherent "co-growth" between detail and hallucination in current OLMs. To address this, we propose Omni-Detective, an agentic data generation pipeline integrating tool-calling, to autonomously produce highly detailed yet minimally hallucinatory multimodal data. Based on the data generated with Omni-Detective, we train two captioning models: Audio-Captioner for audio-only detailed perception, and Omni-Captioner for audio-visual detailed perception. Under the cascade evaluation protocol, Audio-Captioner achieves the best performance on MMAU and MMAR among all open-source models, surpassing Gemini 2.5 Flash and delivering performance comparable to Gemini 2.5 Pro. On existing detailed captioning benchmarks, Omni-Captioner sets a new state-of-the-art on VDC and achieves the best trade-off between detail and hallucination on the video-SALMONN 2 testset. Given the absence of a dedicated benchmark for omni detailed perception, we design Omni-Cloze, a novel cloze-style evaluation for detailed audio, visual, and audio-visual captioning that ensures stable, efficient, and reliable assessment. Experimental results and analysis demonstrate the effectiveness of Omni-Detective in generating high-quality detailed captions, as well as the superiority of Omni-Cloze in evaluating such detailed captions.

Abstract of Omni-Captioner: Data Pipeline, Models, and Benchmark for Omni Detailed Perception (arXiv:2510.12720, CC0)
Made by Ma et al. (Omni-Captioner, arXiv 2510.12720) Released Oct 2025 Video captioning All tasks public Last in a launch post Sep 2026 , dropped before it saturated Last model added Sep 2026
Level
Everyday Most adults, without training
Scored
Checked automatically A program checks the result against an answer key, unit tests or the expected final state.
Answers
Short answer A number, word or short string matched to a key.
2
Models scored in lab cards
2
Launch posts list it, from 1 lab
2
System cards report it, from 1 lab

Qwen3.5-Omni-Plus leads with 64.8%

Best score per model as labs report them in their own cards. Labs pick their own settings, so these compare less cleanly than independent runs.

  1. 1 Qwen3.5-Omni-Plus 64.8%
  2. 2 Qwen3.8-Omni-Flash 63.2%
#ModelScoreSettingDateSource
1 Qwen3.5-Omni-Plus 64.8% Mar 2026 Model card
2 Qwen3.8-Omni-Flash 63.2% Sep 2026 Model card

Number one over time

Lab claims count from the day the card came out, independent runs from the model’s release.

Qwen
Lab claims 1 leader

Scores over time

All the scores we have, one dot each, so a model tested at several settings appears more than once.

Reported by the lab in its model card
0% 20% 40% 60% 80% 100% 2026 Near the ceiling (85%) Qwen3.5-Omni-Plus 0% 20% 40% 60% 80% 100% 2026 Near the ceiling (85%) Qwen3.5-Omni-Plus

Labs headline it less often now, last in Sep 2026

Launch posts from every lab that has listed it, one tick per post. Tall colored ticks include it, and short gray ones leave it out.

Qwen
DateLaunch postScoreAs the post writes it
18 Sep 2026 Qwen3.8-Omni-Flash 63.2 OmniCloze Audio-Visual Caption
30 Mar 2026 Qwen3.5-Omni-Plus 64.8 Omni-Cloze

2 system cards report it, from 1 lab

Besides launch posts, we read the system cards, model cards and technical reports labs publish with a release. They report far more, including safety and frontier-risk tests.

DateDocumentScoreAs the document writes itSection
22 Sep 2026 Qwen3.8-Omni-Flash technical report 63.2 OmniCloze (Table 5) Capabilities
17 Apr 2026 Qwen3.5-Omni-Plus technical report 64.8 Omni-Cloze (Table 7) Capabilities

More in Audio and video

All 168 in Audio and video
  • Video-MMMU In use 300 expert-level lecture videos with questions testing perception, comprehension and adaptation of knowledge. Accuracy. Qwen 3.8 Max 88.7%
  • LVBench In use Hours-long video understanding with multiple-choice questions on reasoning, grounding and summarization. Gemini 4 Argon 91.7%
  • MMVU In use 3,000 expert-annotated questions on videos across 27 disciplines. Multiple-choice and open-ended accuracy. Qwen 3.8 Max 82.4%
  • MLVU In use Multi-task long video understanding (minutes to hours) with multiple-choice and generation tasks. Accuracy. Qwen 3.8 Max 90.8%
  • VoiceBench In use Spoken instructions with speaker, environment and content variation. Voice assistant responses scored by GPT judge or rules. Qwen3.5-Omni-Plus 93.1%
  • TVBench In use Models answer multiple-choice questions on 10 temporal video tasks, such as action counting and ordering, that static frames cannot solve. Scored by accuracy. Qwen 3.8 Max 81.9%