MME-VideoOCR

Models answer 2,000 questions about text that appears in 1,464 videos, across 25 tasks from recognition to reasoning.

Motion blur and visual effects make video OCR harder than image OCR.

In the authors’ words

Multimodal Large Language Models (MLLMs) have achieved considerable accuracy in Optical Character Recognition (OCR) from static images. However, their efficacy in video OCR is significantly diminished due to factors such as motion blur, temporal variations, and visual effects inherent in video content. To provide clearer guidance for training practical MLLMs, we introduce the MME-VideoOCR benchmark, which encompasses a comprehensive range of video OCR application scenarios. MME-VideoOCR features 10 task categories comprising 25 individual tasks and spans 44 diverse scenarios. These tasks extend beyond text recognition to incorporate deeper comprehension and reasoning of textual content within videos. The benchmark consists of 1,464 videos with varying resolutions, aspect ratios, and durations, along with 2,000 meticulously curated, manually annotated question-answer pairs. We evaluate 18 state-of-the-art MLLMs on MME-VideoOCR, revealing that even the best-performing model (Gemini-2.5 Pro) achieves an accuracy of only 73.7%. Fine-grained analysis indicates that while existing MLLMs demonstrate strong performance on tasks where relevant texts are contained within a single or few frames, they exhibit limited capability in effectively handling tasks that demand holistic video comprehension. These limitations are especially evident in scenarios that require spatio-temporal reasoning, cross-frame information integration, or resistance to language prior bias. Our findings also highlight the importance of high-resolution visual input and sufficient temporal coverage for reliable OCR in dynamic video scenarios.

Abstract of MME-VideoOCR: Evaluating OCR-Based Capabilities of Multimodal LLMs in Video Scenarios (arXiv:2505.21333, CC0)

Made by MME team (Shi et al., arXiv 2505.21333) Released May 2025 Video understanding All tasks public Last in a launch post Mar 2026 , dropped before it saturated Last model added Apr 2026

Level: Everyday Most adults, without training
Scored: Checked automatically A program checks the result against an answer key, unit tests or the expected final state.
Answers: Multiple choice

DogNeverSleep/MME-VideoOCR 43 stars Paper

Model scored in lab cards

Launch post lists it, from 1 lab

System card reports it, from 1 lab

Qwen3.5-Omni-Plus leads with 77%

Best score per model as labs report them in their own cards. Labs pick their own settings, so these compare less cleanly than independent runs.

Bars Table

77.0 Qwen3.5-Omni-Plus

1 Qwen3.5-Omni-Plus 77%

#	Model	Score	Setting	Date	Source
1	Qwen3.5-Omni-Plus	77%		Mar 2026	Model card

Scores over time

All the scores we have, one dot each, so a model tested at several settings appears more than once.

Reported by the lab in its model card

Labs headline it less often now, last in Mar 2026

Launch posts from every lab that has listed it, one tick per post. Tall colored ticks include it, and short gray ones leave it out.

Qwen

Date	Launch post	Score	As the post writes it
30 Mar 2026	Qwen3.5-Omni-Plus	77	MME-VideoOCR

One system card reports it

Besides launch posts, we read the system cards, model cards and technical reports labs publish with a release. They report far more, including safety and frontier-risk tests.

Date	Document	Score	As the document writes it	Section
17 Apr 2026	Qwen3.5-Omni-Plus technical report	77	MME-VideoOCR (Table 6)	Capabilities

Qwen3.5-Omni-Plus leads with 77%

Scores over time

Labs headline it less often now, last in Mar 2026

One system card reports it

More in Audio and video