Benchmark Almanac

QMSum

HMNet 36.1 tops its leaderboard
Not in model cards

Models answer a query about a long meeting transcript with a written summary of the relevant parts. The 1,808 query and summary pairs cover 232 meetings from several domains. Scores compare outputs with reference summaries using ROUGE.

SCROLLS and ZeroSCROLLS include QMSum as a task, and the almanac carries SCROLLS as a suite. ROUGE rewards word overlap, not faithfulness. Microsoft reports it in the Phi-3.5 long-context table. About 160 GitHub stars.

Released
April 2021
Kind
Reasoning over long documents
Level
School
Scored
Checked automatically
Answers
Open-ended
Tasks
All public

In the authors’ words

Meetings are a key component of human collaboration. As increasing numbers of meetings are recorded and transcribed, meeting summaries have become essential to remind those who may or may not have attended the meetings about the key decisions made and the tasks to be completed. However, it is hard to create a single short summary that covers all the content of a long meeting involving multiple people and topics. In order to satisfy the needs of different types of users, we define a new query-based multi-domain meeting summarization task, where models have to select and summarize relevant spans of meetings in response to a query, and we introduce QMSum, a new benchmark for this task. QMSum consists of 1,808 query-summary pairs over 232 meetings in multiple domains. Besides, we investigate a locate-then-summarize method and evaluate a set of strong summarization baselines on the task. Experimental results and manual analysis reveal that QMSum presents significant challenges in long meeting summarization for future research. Dataset is available at https://github.com/Yale-LILY/QMSum.

Abstract of QMSum: A New Benchmark for Query-based Multi-domain Meeting Summarization (arXiv:2104.05938, CC0)
4
Entries on its own leaderboard, 0 models we track

HMNet tops the leaderboard with 36.1

Best score per entry from the benchmark's own leaderboard, read on November 12th, 2026. Where a model ran with several settings or harnesses, we keep its best. Names in gray aren't in our model list: fine-tunes, agent systems, products and older models. They don't count toward leaders.

  1. 1 HMNet gold spans 36.1
  2. 2 BART gold spans 32.2
  3. 3 PGNet gold spans 31.5
  4. 4 TextRank full transcript 16.3

Source: Official QMSum leaderboard

#EntryScoreSettingDateSource
1 HMNet 36.1 gold spans Official leaderboard
2 BART 32.2 gold spans Official leaderboard
3 PGNet 31.5 gold spans Official leaderboard
4 TextRank 16.3 full transcript Official leaderboard

No lab model card reports it yet

We read the headline results of 522 launch posts from 52 labs, and none of them list this benchmark.

More in Long context

All 79 in Long context
  • MRCR v2 In use Multi-round coreference: retrieve the n-th of several similar requests hidden in a long synthetic conversation, up to 1M tokens. Muse Spark 1.3 98.5%
  • LongBench v2 In use 503 multiple-choice questions over 8K-2M-word contexts (QA, dialogue, code repos, structured data). Accuracy. Inkling 2 Small 70.6%
  • GraphWalks In use A graph is given as an edge list in long context. The model answers BFS or parent queries, scored by F1. Claude Fable 5 99.96%
  • OpenAI MRCR v2 In use Models read a long synthetic chat that hides 2, 4 or 8 identical writing requests and must reproduce the response to the i-th one. Scored by mean string match ratio per context bin. Gemini 3.7 Flash 89%
  • CL-bench In use Model must learn new knowledge from a supplied long context (rules, procedures, data) and solve tasks. Strict LLM-verified rubrics. GPT-5.4 27.9%
  • AA-LCR 1.1 In use Models answer 100 questions that need reasoning across sets of real documents of about 100K tokens. An LLM equality checker with a fixed system prompt compares each answer with the key. Step 5 Preview 88.3%