VoiceBench
Qwen3.5-Omni-Plus 93.1% leads since Mar 2026 In use Spoken instructions with speaker, environment and content variation. Voice assistant responses scored by GPT judge or rules.
The suite grew after release: mmsu joined the leaderboard in 2024-12, and bbh and wildvoice in 2025-04. The Overall score now averages nine subsets, so early Overall scores are not comparable.
In the authors’ words
Building on the success of large language models (LLMs), recent advancements such as GPT-4o have enabled real-time speech interactions through LLM-based voice assistants, offering a significantly improved user experience compared to traditional text-based interactions. However, the absence of benchmarks designed to evaluate these speech interaction capabilities has hindered progress of LLM-based voice assistants development. Current evaluations focus primarily on automatic speech recognition (ASR) or general knowledge evaluation with clean speeches, neglecting the more intricate, real-world scenarios that involve diverse speaker characteristics, environmental and content factors. To address this, we introduce VoiceBench, the first benchmark designed to provide a multi-faceted evaluation of LLM-based voice assistants. VoiceBench also includes both real and synthetic spoken instructions that incorporate the above three key real-world variations. Extensive experiments reveal the limitations of current LLM-based voice assistant models and offer valuable insights for future research and development in this field.
- Level
- Everyday Most adults, without training
- Scored
- Graded by a model A language model scores each response, usually against a rubric.
- Answers
- Open-ended Free-form text, documents, designs or media judged as a whole.
Qwen3.5-Omni-Plus leads with 93.1%
Best score per model as labs report them in their own cards. Labs pick their own settings, so these compare less cleanly than independent runs.
- 1 Qwen3.5-Omni-Plus 93.1%
- 2 Qwen3.8-Omni-Flash 91.6%
- 3
Inkling 91.4% - 4 Inkling-Small 90.1%
- 5
Nemotron 3 Nano Omni 30B-A3B 89.4% - 6
LongCat-Flash-Omni 88.7% - 7 Qwen3-Omni-30B-A3B 85.5%
- 8 Nemotron 3.5 Bolt E4B 80.3%
- 9 Kimi-Audio-7B-Instruct 76.9%
- 10 Qwen2.5-Omni-7B 74.1%
| # | Model | Score | Setting | Date | Source |
|---|---|---|---|---|---|
| 1 | Qwen3.5-Omni-Plus | 93.1% | Mar 2026 | Model card | |
| 2 | Qwen3.8-Omni-Flash | 91.6% | Sep 2026 | Model card | |
| 3 | Inkling | 91.4% | Jul 2026 | Model card | |
| 4 | Inkling-Small | 90.1% | Jul 2026 | Model card | |
| 5 | Nemotron 3 Nano Omni 30B-A3B | 89.4% | Apr 2026 | Model card | |
| 6 | LongCat-Flash-Omni | 88.7% | Oct 2025 | Model card | |
| 7 | Qwen3-Omni-30B-A3B | 85.5% | Sep 2025 | Model card | |
| 8 | Nemotron 3.5 Bolt E4B | 80.3% | Oct 2026 | Model card | |
| 9 | Kimi-Audio-7B-Instruct | 76.9% | Apr 2025 | Model card | |
| 10 | Qwen2.5-Omni-7B | 74.1% | Mar 2025 | Model card |
The benchmark's own leaderboard has 41 entries, 15 of them models we track. We plot every source in the charts below.
8 models have held number one since its release in Oct 2024
Lab claims count from the day the card came out, independent runs from the model’s release. We draw one row each for the benchmark’s own leaderboard and what labs claim in their cards.
The best score passed 85% in Sep 2025
All the scores we have, one dot each, so a model tested at several settings appears more than once.
2 labs put it in a launch post in the last six months
Launch posts from every lab that has listed it, one tick per post. Tall colored ticks include it, and short gray ones leave it out.
| Date | Launch post | Score | As the post writes it |
|---|---|---|---|
| 18 Sep 2026 | Qwen3.8-Omni-Flash | 91.6 | VoiceBench Audio Interaction |
| 30 Jul 2026 | Inkling-Small | 90.1 | VoiceBench |
| 15 Jul 2026 | Inkling | 91.4 | VoiceBench |
| 28 Apr 2026 | Nemotron 3 Nano Omni 30B-A3B | 89.4 | Voice interaction (Speech Instruction Following) |
| 30 Mar 2026 | Qwen3.5-Omni-Plus | 93.1 | VoiceBench |
| 31 Oct 2025 | LongCat-Flash-Omni | 88.7 | Mean of VoiceBench AlpacaEval 4.94 / CommonEval 4.32 (GPT scores x20) / OpenBookQA 93.41 / SDQA 82.46 / MMSU 81.95 / AdvBench 100 / IFEval 77.99, computed by the almanac |
| 22 Sep 2025 | Qwen3-Omni-30B-A3B | 85.5 | VoiceBench Overall |
| 25 Apr 2025 | Kimi-Audio-7B-Instruct | 76.9 | VoiceBench Avg (AlpacaEval 4.46, CommonEval 3.97, SD-QA 63.12, MMSU 62.17, OpenBookQA 83.52, IFEval 61.10, AdvBench 100.00) |
| 27 Mar 2025 | Qwen2.5-Omni-7B | 74.1 | VoiceBench Avg |
12 system cards report it, from 7 labs
Besides launch posts, we read the system cards, model cards and technical reports labs publish with a release. They report far more, including safety and frontier-risk tests.
| Date | Document | Score | As the document writes it | Section |
|---|---|---|---|---|
| 23 Oct 2026 | Nemotron 3.5 Bolt E4B technical report | 80.3 | Voicebench, reasoning, 72.60 non-reasoning (Table 8) | Capabilities |
| 22 Sep 2026 | Qwen3.8-Omni-Flash technical report | 91.6 | VoiceBench (Table 4) | Capabilities |
| 30 Jul 2026 | Inkling-Small model card | 90.1 | VoiceBench (5. Evaluations table) | Capabilities |
| 15 Jul 2026 | Inkling model card | 91.4 | VoiceBench (5. Evaluations table, effort 0.99) | Capabilities |
| 27 Apr 2026 | Nemotron 3 Nano Omni 30B-A3B technical report | 89.4 | VoiceBench Avg, reasoning on (Table 8). Nine spoken subsets from 71.4 (SD-QA) to 100 (AdvBench) | Capabilities |
| 17 Apr 2026 | Qwen3.5-Omni-Plus technical report | 93.1 | VoiceBench (Table 5) | Capabilities |
| 4 Feb 2026 |  ERNIE 5.0 technical report | VoiceBench AlpacaEval 4.62, CommonEval 3.74, SD-QA 77.58, MMSU 84.68, OpenBookQA 92.97, IFEval 72.67, AdvBench 99.23, no overall score (Table 7) | Capabilities | |
| 31 Oct 2025 | LongCat-Flash-Omni technical report | VoiceBench AlpacaEval 4.94, CommonEval 4.32, OpenBookQA 93.41, SDQA 82.46, MMSU 81.95, AdvBench 100, IFEval 77.99, no overall score (Table 11) | Capabilities | |
| 22 Sep 2025 | Qwen3-Omni-30B-A3B technical report | 85.5 | VoiceBench Overall, Instruct (Table 7). Thinking 88.8 | Capabilities |
| 17 Jul 2025 |  Voxtral Small technical report | Openbook QA (Chen et al., 2024, the VoiceBench subset) 88.4, no overall VoiceBench score (Table 8) | Capabilities | |
| 25 Apr 2025 | Kimi-Audio-7B-Instruct technical report | 76.9 | VoiceBench Avg (Table 6). AlpacaEval 4.46, CommonEval 3.97, SD-QA 63.12, MMSU 62.17, OpenBookQA 83.52, IFEval 61.10, AdvBench 100.00 | Capabilities |
| 26 Mar 2025 | Qwen2.5-Omni-7B technical report | 74.1 | VoiceBench Avg (Table 3). AlpacaEval 4.49, CommonEval 3.93, SD-QA 55.71, MMSU 61.32, OpenBookQA 81.10, IFEval 52.87, AdvBench 99.42 | Capabilities |
More in Audio and video
- Video-MMMU In use 300 expert-level lecture videos with questions testing perception, comprehension and adaptation of knowledge. Accuracy. Qwen 3.8 Max 88.7%
- LVBench In use Hours-long video understanding with multiple-choice questions on reasoning, grounding and summarization.
Gemini 4 Argon 91.7% - MMVU In use 3,000 expert-annotated questions on videos across 27 disciplines. Multiple-choice and open-ended accuracy. Qwen 3.8 Max 82.4%
- MLVU In use Multi-task long video understanding (minutes to hours) with multiple-choice and generation tasks. Accuracy. Qwen 3.8 Max 90.8%
- TVBench In use Models answer multiple-choice questions on 10 temporal video tasks, such as action counting and ordering, that static frames cannot solve. Scored by accuracy. Qwen 3.8 Max 81.9%
- VideoMME v2 (with subtitles) In use Models answer multiple choice questions about videos with subtitles or audio added. Related questions are scored as groups to penalize guessing. Qwen 3.8 Max 68.3%