Benchmark Almanac

SpatialTree

Only in system cards

Organizes spatial abilities into a hierarchy, from low-level perception to mental mapping, simulation and agentic use, and tests vision-language models at each level.

Made by
Xiao et al.
Released
December 2025
Kind
Spatial reasoning and counting
Level
Everyday
Scored
Checked automatically
Answers
Short answer
Tasks
All public
Last in a system card
October 2026
Last model added
October 2026

In the authors’ words

Cognitive science suggests that spatial ability develops progressively-from perception to reasoning and interaction. Yet in multimodal LLMs (MLLMs), this hierarchy remains poorly understood, as most studies focus on a narrow set of tasks. We introduce SpatialTree, a cognitive-science-inspired hierarchy that organizes spatial abilities into four levels: low-level perception (L1), mental mapping (L2), simulation (L3), and agentic competence (L4). Based on this taxonomy, we construct the first capability-centric hierarchical benchmark, thoroughly evaluating mainstream MLLMs across 27 sub-abilities. The evaluation results reveal a clear structure: L1 skills are largely orthogonal, whereas higher-level skills are strongly correlated, indicating increasing interdependency. Through targeted supervised fine-tuning, we uncover a surprising transfer dynamic-negative transfer within L1, but strong cross-level transfer from low- to high-level abilities with notable synergy. Finally, we explore how to improve the entire hierarchy. We find that naive RL that encourages extensive "thinking" is unreliable: it helps complex reasoning but hurts intuitive perception. We propose a simple auto-think strategy that suppresses unnecessary deliberation, enabling RL to consistently improve performance across all levels. By building SpatialTree, we provide a proof-of-concept framework for understanding and systematically scaling spatial abilities in MLLMs.

Abstract of SpatialTree: How Spatial Abilities Branch Out in MLLMs (arXiv:2512.20617, CC0)
1
Model scored in lab cards
1
System card reports it, from 1 lab

Nemotron 3.5 Super VL leads with 58.1%

Best score per model as labs report them in their own cards. Labs pick their own settings, so these compare less cleanly than independent runs.

  1. 1 Nemotron 3.5 Super VL 58.1%

Source: Labs’ model cards

#ModelScoreSettingDateSource
1 Nemotron 3.5 Super VL 58.1% Oct 2026 Lab model card

Scores over time

All the scores we have, one dot each, so a model tested at several settings appears more than once.

Reported by the lab in its model card
0% 20% 40% 60% 80% 100% Near the ceiling (85%) Nemotron 3.5 Super VL 0% 20% 40% 60% 80% 100% Near the ceiling (85%) Nemotron 3.5 Super VL

Source: Labs’ model cards

Labs report it only in system cards

We read the headline results of 522 launch posts from 52 labs, and none of them list this benchmark. Some system cards do, below.

One system card reports it

Besides launch posts, we read the system cards, model cards and technical reports labs publish with a release. They report far more, including safety and frontier-risk tests.

DateDocumentScoreAs the document writes itSection
Oct 15, 2026 Nemotron 3.5 Super VL technical report 58.1 SpatialTree, SpatialClaw harness, with agentic vision training, Table 7 Capabilities

More in Vision

All 281 in Vision
  • MMMU In use 11.5K college-level multimodal questions across 30 subjects. Multiple-choice and open accuracy. GPT-5.1 85.4%
  • MMMU-Pro In use Harder MMMU: college-level multimodal questions with more options and vision-only inputs to remove text shortcuts. GPT-5.6 Sol 84.6%
  • CharXiv In use Chart understanding on real arXiv figures. Reasoning questions require synthesizing information from complex charts. Claude Fable 5 93.5%
  • MATH-Vision (MathVision) In use 3,040 competition math problems with visual context. Answer accuracy. Qwen 3.8 Max 97.7%
  • MMStar In use Models answer 1,500 human-curated image questions that cannot be solved without the image, across 6 core capabilities. Scored by accuracy. Qwen 3.8 Max 85.9%
  • ChartQA In use Models answer human-written and generated questions about chart images that need visual and arithmetic reasoning. Scored by relaxed accuracy with a 5 percent numeric tolerance. Command A Vision 90.9%