- MMMUval 70.2
- MMMU_Pro 51.1
- MathVista_MINI 74.8
- MathVision_FULL 38.1
- Hallusion Bench 55.2
- MMBench_DEV_EN_V11 88
- AI2D_TEST 88.4
- ChartQA_TEST 89.5
- DocVQA_VAL 96.4
- MMStar 70.8
- MMVet_turbo 76.2
- OCRBench 885 (out of 1000) 88.5
- OCRBench-V2(en/zh) 61.5/63.7 61.5
- CC-OCR 79.8
- Best of VideoMME w sub. 79.1 / VideoMME w/o sub. 73.3 79.1
- MVBench 70.4
- MMBench-Video 2.02
- LVBench 47.3
- EgoSchema 76.2
- PerceptionTest_test 73.2
- MLVU_M-Avg_dev 74.6
- TempCompass_overall 74.8
- ScreenSpot 87.1
- ScreenSpot Pro 43.6
- AITZ_EM 83.2
- Android Control High_EM (Low_EM 93.7) 67.4
- AndroidWorld_SR 35
- MobileMiniWob++_SR 68
- OSWorld 8.83
Qwen2.5-VL-72B
Qwen2.5-VL-72B ranks first on 2 of the 71 benchmarks it has scores on.
81
Benchmarks with a score
2
Ranked first, out of five or more models
10
In the top three
Qwen Jan 2025 Qwen2.5-VL-72B
Open weights Flagship
No Epoch Capabilities Index score yet
- MMMU-Pro 51.1% #65 /72
- MMVU 62.9% #18 /20
- HallusionBench 55.2% #22 /23
Where it ranks
Its best score on each benchmark, and where that ranks among every model with a score there. We rank each model once, at its best setting.
Knowledge and language
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| MMBench-CN v1.1 | 87.4% | #5 of 178 | Official leaderboard | |
| Creation-MMBench | -5.82 | #4 of 19 | Official leaderboard | |
| IFEval | 86.3% | #61 of 126 | Model card | |
| MMLU-Pro | 71.2% | #109 of 151 | Model card | |
| MMLU-Redux | 85.9% | #30 of 38 | Model card | |
| HallusionBench | 55.2% | #22 of 23 | Model card |
Math and science
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| GSM8K | 95.3% | #11 of 87 | Model card | |
| MATH | 83% | #17 of 93 | Model card | |
| HiPhO | 29% | #29 of 35 | Official leaderboard | |
| GPQA Diamond | 49% | #236 of 284 | Model card |
Coding
Agents and tools
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| ScreenSpot | 87.1% | #7 of 11 | Model card | |
| AndroidWorld | 62.9% | #7 of 10 | Official leaderboard | |
| SnakeBench | 20 | #131 of 183 | Official leaderboard | |
| ScreenSpot-Pro | 43.6% | #23 of 25 | Model card | |
| OSWorld-Verified | 5.00% | #32 of 33 | Official leaderboard | |
| OSWorld | 8.83% | #12 of 12 | Model card | |
| AndroidControl | 67.4% | #2 of 2 | Model card | |
| Android in the Zoo (AitZ) | 83.2% | #1 of 1 | Model card | |
| MobileMiniWoB++ | 68% | #1 of 1 | Model card |
Professional work
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| MedXpertQA-MM | 29.9% | #7 of 9 | Official leaderboard |
Vision, audio and robotics
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| A-Bench | 81% | #1 of 116 | Official leaderboard | |
| MMBench-EN v1.1 | 88.3% | #2 of 178 | Official leaderboard | |
| MMBench | 88.3% | #3 of 155 | Official leaderboard | |
| VHELM | 78.0 | #3 of 41 | Official leaderboard | |
| EgoSchema | 76.2% | #1 of 12 | Model card | |
| DocVQA | 96.4% | #3 of 33 | Model card | |
| CCBench (MMBench Chinese Culture) | 73.9% | #22 of 152 | Official leaderboard | |
| ChartQA | 89.5% | #8 of 37 | Model card | |
| CRPE | 79.2% | #1 of 4 | Model card | |
| OCRBench | 88.5% | #8 of 28 | Model card | |
| TextVQA | 83.5% | #5 of 17 | Model card | |
| InfographicVQA | 87.3% | #7 of 22 | Model card | |
| MME | 2448.0 | #2 of 6 | Model card | |
| SEED-Bench-2-Plus | 73% | #1 of 3 | Model card | |
| MMMU | 64.5% | #48 of 136 | Official leaderboard | |
| Point-Bench | 59% | #6 of 17 | Official leaderboard | |
| MM-Vet | 76.2% | #3 of 8 | Model card | |
| VisualPuzzles | 42.2% | #12 of 29 | Official leaderboard | |
| MathVista | 74.8% | #32 of 64 | Model card | |
| AI2D | 88.7% | #22 of 44 | Model card | |
| BLINK | 64.4% | #14 of 28 | Model card | |
| MM-MT-Bench | 7.60 | #3 of 6 | Model card | |
| Perception Test MCVQA | 73.2% | #3 of 6 | Model card | |
| MME-RealWorld | 63.2% | #2 of 4 | Model card | |
| MMBench-Video | 2.02 | #1 of 2 | Model card | |
| VideoMME v2 (with subtitles) | 22.1% | #16 of 31 | Official leaderboard | |
| CountBench | 93.6% | #11 of 20 | Model card | |
| RealWorldQA | 75.7% | #17 of 30 | Model card | |
| Video-MME | 79.1% | #24 of 42 | Model card | |
| OCRBench v2 | 61.5% | #12 of 20 | Model card | |
| VisuLogic | 26.2% | #11 of 17 | Official leaderboard | |
| MuirBench | 70.7% | #8 of 12 | Model card | |
| MTVQA-Test | 31.7% | #4 of 6 | Model card | |
| VCR (Visual Caption Restoration) | 79.8% | #2 of 3 | Model card | |
| RoboRewardBench | 0.99 | #13 of 19 | Official leaderboard | |
| OmniDocBench v1.5 | 87.0 | #7 of 10 | Official leaderboard | |
| GeoBench | 62% | #34 of 48 | Official leaderboard | |
| MMStar | 70.8% | #23 of 30 | Model card | |
| Math-VR | 13.7% | #20 of 26 | Official leaderboard | |
| Vision Arena | 1,121 | #110 of 136 | Official leaderboard | |
| CharXiv (DQ) | 87.4% | #6 of 7 | Model card | |
| MVBench | 70.4% | #13 of 15 | Model card | |
| MLVU | 74.6% | #14 of 16 | Model card | |
| ODinW-13 | 43.1% | #8 of 9 | Model card | |
| MMVU | 62.9% | #18 of 20 | Model card | |
| MMMU-Pro | 51.1% | #65 of 72 | Model card | |
| LongVideoBench | 60.7% | #10 of 11 | Model card | |
| MathVerse | 57.6% | #11 of 12 | Model card | |
| MATH-Vision (MathVision) | 38.1% | #39 of 42 | Model card | |
| CharXiv | 49.7% | #63 of 67 | Model card | |
| ClockBench | 6.10% | #35 of 37 | Official leaderboard | |
| LVBench | 47.3% | #24 of 25 | Model card | |
| Video-MMMU | 60.2% | #30 of 30 | Model card | |
| CC-OCR | 79.8% | #8 of 8 | Model card | |
| TempCompass | 74.8% | #7 of 7 | Model card | |
| Charades-STA | 50.9% | #3 of 3 | Model card | |
| MEGA-Bench | 51.3% | #1 of 1 | Model card |
Other
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| OpenVLM Leaderboard | 76.1 | #8 of 173 | Official leaderboard | |
| LiveBench | 57% | #17 of 27 | Model card |
Its launch post
The benchmarks the lab chose to headline, with the scores it gave.
Its system card
Every result in the longer documents the lab published with it. 34 of them aren't in the launch post.
Qwen2.5-VL Technical Report 19 Feb 2025
Capabilities
- MMMU 70.2
- MMMU-Pro 51.1
- MathVista 74.8
- MATH-Vision (MathVision) 38.1
- MathVerse 57.6
- MEGA-Bench 51.3
- MMBench 88.6
- MMBench-EN v1.1 88.4
- MMStar 70.8
- MME 2448
- MuirBench 70.7
- BLINK 64.4
- CRPE 79.2
- HallusionBench 55.2
- MTVQA-Test 31.7
- RealWorldQA 75.7
- MME-RealWorld 63.2
- MM-Vet 76.2
- MM-MT-Bench 7.6
- MMLU-Pro 71.2
- MMLU-Redux 85.9
- LiveBench 57
- GPQA Diamond 49
- MATH 83
- GSM8K 95.3
- HumanEval 87.8
- MultiPL-E 79.5
- IFEval 86.3
- CC-OCR 79.8
- OmniDocBench
- AI2D 88.7
- TextVQA 83.5
- DocVQA 96.4
- InfographicVQA 87.3
- ChartQA 89.5
- CharXiv 49.7
- CharXiv (DQ) 87.4
- SEED-Bench-2-Plus 73
- OCRBench 88.5
- VCR (Visual Caption Restoration) 79.8
- OCRBench v2
- RefCOCO / RefCOCO+ / RefCOCOg
- ODinW-13 43.1
- CountBench 93.6
- Video-MME 73.3
- Video-MMMU 60.2
- MMVU 62.9
- MVBench 70.4
- MMBench-Video 2.02
- LongVideoBench 60.7
- LVBench 47.3
- EgoSchema 76.2
- Perception Test MCVQA 73.2
- MLVU 74.6
- TempCompass 74.8
- Charades-STA 50.9
- ScreenSpot 87.1
- ScreenSpot-Pro 43.6
- AndroidControl 67.4
- AndroidWorld 35
- MobileMiniWoB++ 68
- OSWorld 8.83