Benchmark Almanac

MusTBench

Models answer five kinds of music questions that need the correct time region of the audio, such as instrument entries. Music experts validated the items.

Qwen reports it as music understanding.

In the authors’ words

Recent Large Audio-Language Models (LALMs) have demonstrated promising abilities in understanding musical content. However, whether their responses are grounded in the correct temporal regions of the audio remains underexplored. This limitation is particularly critical for music understanding, where key information often occurs as temporally localized events, such as instrument entries and rhythmic transitions. To address this gap, we introduce MusTBench, a music-expert-validated benchmark designed to evaluate temporal grounding in LALMs through five temporally grounded question-answering tasks. To further improve temporal grounding in existing models, we propose MusT, a novel four-stage temporal optimization recipe spanning music encoder adaptation, LLM adaptation, LLM supervised fine-tuning, and RL-based optimization. Experiments on MusTBench show that existing LALMs struggle with precise temporal grounding, while MusT brings significant improvements over strong baselines. These results establish temporal grounding as a key missing capability in current LALMs and position MusTBench as a challenging benchmark for future research in temporally grounded music understanding.

Abstract of MusTBench: Benchmarking and Advancing Temporal Grounding in Music LLMs (arXiv:2605.29300, CC0)
Made by Kwon et al. (arXiv 2605.29300) Released May 2026 Audio understanding and voice All tasks public Last in a launch post Sep 2026 Last model added Sep 2026
Level
Expert Specialists with years of training, often over hours of work
Scored
Checked automatically A program checks the result against an answer key, unit tests or the expected final state.
Answers
Short answer A number, word or short string matched to a key.
1
Model scored in lab cards
1
Launch post lists it, from 1 lab
1
System card reports it, from 1 lab

Qwen3.8-Omni-Flash leads with 50.6%

Best score per model as labs report them in their own cards. Labs pick their own settings, so these compare less cleanly than independent runs.

  1. 1 Qwen3.8-Omni-Flash 50.6%
#ModelScoreSettingDateSource
1 Qwen3.8-Omni-Flash 50.6% Sep 2026 Model card

Scores over time

All the scores we have, one dot each, so a model tested at several settings appears more than once.

Reported by the lab in its model card
0% 20% 40% 60% 80% 100% Near the ceiling (85%) Qwen3.8-Omni-Flash 0% 20% 40% 60% 80% 100% Near the ceiling (85%) Qwen3.8-Omni-Flash

One lab put it in a launch post in the last six months

Launch posts from every lab that has listed it, one tick per post. Tall colored ticks include it, and short gray ones leave it out.

Qwen
DateLaunch postScoreAs the post writes it
18 Sep 2026 Qwen3.8-Omni-Flash 50.6 MusTBench Music Understanding

One system card reports it

Besides launch posts, we read the system cards, model cards and technical reports labs publish with a release. They report far more, including safety and frontier-risk tests.

DateDocumentScoreAs the document writes itSection
22 Sep 2026 Qwen3.8-Omni-Flash technical report 50.6 MusTBench (Table 4) Capabilities

More in Audio and video

All 168 in Audio and video
  • Video-MMMU In use 300 expert-level lecture videos with questions testing perception, comprehension and adaptation of knowledge. Accuracy. Qwen 3.8 Max 88.7%
  • LVBench In use Hours-long video understanding with multiple-choice questions on reasoning, grounding and summarization. Gemini 4 Argon 91.7%
  • MMVU In use 3,000 expert-annotated questions on videos across 27 disciplines. Multiple-choice and open-ended accuracy. Qwen 3.8 Max 82.4%
  • MLVU In use Multi-task long video understanding (minutes to hours) with multiple-choice and generation tasks. Accuracy. Qwen 3.8 Max 90.8%
  • VoiceBench In use Spoken instructions with speaker, environment and content variation. Voice assistant responses scored by GPT judge or rules. Qwen3.5-Omni-Plus 93.1%
  • TVBench In use Models answer multiple-choice questions on 10 temporal video tasks, such as action counting and ordering, that static frames cannot solve. Scored by accuracy. Qwen 3.8 Max 81.9%