Benchmark Almanac

VSR Random Split

LXMERT 70.1% tops its leaderboard
Not in model cards

Models judge whether image captions correctly describe spatial relations on the randomly partitioned test set. The score is accuracy over 2,195 test examples.

One track of VSR. The VSR record keeps lab scores that combine tracks.

Released
April 2022
Kind
Spatial reasoning and counting
Level
Everyday
Scored
Checked automatically
Answers
Multiple choice
Tasks
All public

In the authors’ words

Spatial relations are a basic part of human cognition. However, they are expressed in natural language in a variety of ways, and previous work has suggested that current vision-and-language models (VLMs) struggle to capture relational information. In this paper, we present Visual Spatial Reasoning (VSR), a dataset containing more than 10k natural text-image pairs with 66 types of spatial relations in English (such as: under, in front of, and facing). While using a seemingly simple annotation format, we show how the dataset includes challenging linguistic phenomena, such as varying reference frames. We demonstrate a large gap between human and model performance: the human ceiling is above 95%, while state-of-the-art models only achieve around 70%. We observe that VLMs' by-relation performances have little correlation with the number of training examples and the tested models are in general incapable of recognising relations concerning the orientations of objects.

Abstract of Visual Spatial Reasoning (arXiv:2205.00363, CC0)
5
Entries on its own leaderboard, 0 models we track

LXMERT tops the leaderboard with 70.1%

Best score per entry from the benchmark's own leaderboard, read on November 12th, 2026. Where a model ran with several settings or harnesses, we keep its best. Names in gray aren't in our model list: fine-tunes, agent systems, products and older models. They don't count toward leaders.

  1. 70.1 LXMERT
  2. 69.3 ViLT
  3. 65.1 CLIP (finetuned)
  4. 56.0 CLIP (frozen)
  5. 55.2 VisualBERT
  1. 1 LXMERT 70.1%
  2. 2 ViLT 69.3%
  3. 3 CLIP (finetuned) 65.1%
  4. 4 CLIP (frozen) 56%
  5. 5 VisualBERT 55.2%

Source: Official VSR Random Split leaderboard

#EntryScoreSettingDateSource
1 LXMERT 70.1% Official leaderboard
2 ViLT 69.3% Official leaderboard
3 CLIP (finetuned) 65.1% Official leaderboard
4 CLIP (frozen) 56% Official leaderboard
5 VisualBERT 55.2% Official leaderboard

No lab model card reports it yet

We read the headline results of 522 launch posts from 52 labs, and none of them list this benchmark.

More in Vision

All 281 in Vision
  • MMMU In use 11.5K college-level multimodal questions across 30 subjects. Multiple-choice and open accuracy. GPT-5.1 85.4%
  • MMMU-Pro In use Harder MMMU: college-level multimodal questions with more options and vision-only inputs to remove text shortcuts. GPT-5.6 Sol 84.6%
  • CharXiv In use Chart understanding on real arXiv figures. Reasoning questions require synthesizing information from complex charts. Claude Fable 5 93.5%
  • MATH-Vision (MathVision) In use 3,040 competition math problems with visual context. Answer accuracy. Qwen 3.8 Max 97.7%
  • MMStar In use Models answer 1,500 human-curated image questions that cannot be solved without the image, across 6 core capabilities. Scored by accuracy. Qwen 3.8 Max 85.9%
  • ChartQA In use Models answer human-written and generated questions about chart images that need visual and arithmetic reasoning. Scored by relaxed accuracy with a 5 percent numeric tolerance. Command A Vision 90.9%