QMSum
Models answer a query about a long meeting transcript with a written summary of the relevant parts. The 1,808 query and summary pairs cover 232 meetings from several domains. Scores compare outputs with reference summaries using ROUGE.
SCROLLS and ZeroSCROLLS include QMSum as a task, and the almanac carries SCROLLS as a suite. ROUGE rewards word overlap, not faithfulness. Microsoft reports it in the Phi-3.5 long-context table. About 160 GitHub stars.
- Made by
- Yale University (LILY Lab) and collaborators
- Released
- April 2021
- Kind
- Reasoning over long documents
- Level
- School
- Scored
- Checked automatically
- Answers
- Open-ended
- Tasks
- All public
In the authors’ words
Meetings are a key component of human collaboration. As increasing numbers of meetings are recorded and transcribed, meeting summaries have become essential to remind those who may or may not have attended the meetings about the key decisions made and the tasks to be completed. However, it is hard to create a single short summary that covers all the content of a long meeting involving multiple people and topics. In order to satisfy the needs of different types of users, we define a new query-based multi-domain meeting summarization task, where models have to select and summarize relevant spans of meetings in response to a query, and we introduce QMSum, a new benchmark for this task. QMSum consists of 1,808 query-summary pairs over 232 meetings in multiple domains. Besides, we investigate a locate-then-summarize method and evaluate a set of strong summarization baselines on the task. Experimental results and manual analysis reveal that QMSum presents significant challenges in long meeting summarization for future research. Dataset is available at https://github.com/Yale-LILY/QMSum.
HMNet tops the leaderboard with 36.1
Best score per entry from the benchmark's own leaderboard, read on November 12th, 2026. Where a model ran with several settings or harnesses, we keep its best. Names in gray aren't in our model list: fine-tunes, agent systems, products and older models. They don't count toward leaders.
- 1 HMNet gold spans 36.1
- 2 BART gold spans 32.2
- 3 PGNet gold spans 31.5
- 4 TextRank full transcript 16.3
Source: Official QMSum leaderboard
| # | Entry | Score | Setting | Date | Source |
|---|---|---|---|---|---|
| 1 | HMNet | 36.1 | gold spans | Official leaderboard | |
| 2 | BART | 32.2 | gold spans | Official leaderboard | |
| 3 | PGNet | 31.5 | gold spans | Official leaderboard | |
| 4 | TextRank | 16.3 | full transcript | Official leaderboard |
No lab model card reports it yet
We read the headline results of 522 launch posts from 52 labs, and none of them list this benchmark.
More in Long context
- MRCR v2 In use Multi-round coreference: retrieve the n-th of several similar requests hidden in a long synthetic conversation, up to 1M tokens.
Muse Spark 1.3 98.5%
- LongBench v2 In use 503 multiple-choice questions over 8K-2M-word contexts (QA, dialogue, code repos, structured data). Accuracy.
Inkling 2 Small 70.6%
- GraphWalks In use A graph is given as an edge list in long context. The model answers BFS or parent queries, scored by F1. Claude Fable 5 99.96%
- OpenAI MRCR v2 In use Models read a long synthetic chat that hides 2, 4 or 8 identical writing requests and must reproduce the response to the i-th one. Scored by mean string match ratio per context bin.
Gemini 3.7 Flash 89%
- CL-bench In use Model must learn new knowledge from a supplied long context (rules, procedures, data) and solve tasks. Strict LLM-verified rubrics. GPT-5.4 27.9%
- AA-LCR 1.1 In use Models answer 100 questions that need reasoning across sets of real documents of about 100K tokens. An LLM equality checker with a fixed system prompt compares each answer with the key.
Step 5 Preview 88.3%