Benchmark Almanac

RecreationBench

GPT-6 Astra 58.1% leads since Sep 2026 In use

Agents explore a running reference app and rebuild it, across 250 tasks on five platforms. Programmatic tests and a VLM judge score the result.

In the authors’ words

Computer-use agents (CUAs) have advanced along two separate lines: graphical interaction and software development through code and the command line. Real digital work requires both, interleaved rather than stacked end to end. We study hybrid CUAs that autonomously decide when to explore an interface, implement software, and run and visually verify their artifacts. We introduce RecreationWorld, a five-platform framework built around recreation: given a running reference, an agent must discover its behavior and build a faithful implementation with no prescribed workflow. RecreationWorld provides reproducible environments on Ubuntu, macOS, Windows, Android, and Web, plus a unified harness with native GUI control and coding tools. The running reference serves as an oracle for hidden behavioral tests, providing execution-grounded rewards. We scale trajectory generation with high-quality open-source applications. Models trained on these trajectories improve across five out-of-distribution coding and hybrid computer-use benchmarks and more frequently verify their rendered outputs, providing evidence of transfer beyond recreation. For held-out evaluation, we introduce RecreationBench, comprising 250 diverse tasks across domains and platforms. Reference-grounded programmatic and visual assertions cover action-conditioned outcomes at multiple interaction depths; each is validated on the reference and by human reviewers before the suite is frozen for automatic scoring. GPT-6 Astra leads at 58.1% overall, but passes all programmatic tests on just 2.8% of tasks. Agents reproduce static interface structure more reliably than interactions and computed outputs, while generated applications remain smaller and more monolithic than their references. We release the benchmark, environments, and test suites.

Abstract of RecreationWorld: Scalable and Verifiable Environments for Hybrid Computer-Use Agents (arXiv:2609.22000, CC0)
Made by Qwen Released Sep 2026 Desktop apps All tasks public Last in a launch post Aug 2026 Last model added Sep 2026 Scores refreshed 2 Nov 2026
Level
Expert Specialists with years of training, often over hours of work
Scored
Checked automatically A program checks the result against an answer key, unit tests or the expected final state.
Answers
Agent task Many steps in a terminal, browser, desktop, app, game or robot.
10
Models scored on its own leaderboard
3
Launch posts list it, from 1 lab

GPT-6 Astra leads with 58.1% among models released by Sep 2026

Best score per model from the benchmark's own leaderboard, read on 2 Nov 2026. Where a model ran with several settings or harnesses, we keep its best. Every run is in the next chart.

The newest model with a score here came out in Sep 2026, and nobody has tested a newer one. Among the 20 highest-ranked models on the capability index, these have no score yet:

  1. 1 GPT-6 Astra max 58.1%
  2. 2 Claude Opus 5 max 44.2%
  3. 3 GPT-5.6 Sol max 42.1%
  4. 4 Grok 4.6 xhigh 36.7%
  5. 5 Qwen3.8 Max (0902) xhigh 34.8%
  6. 6 Kimi K3 max 31.4%
  7. 7 Claude Opus 4.8 max 31.1%
  8. 8 GLM-5.3 max 24.4%
  9. 9 Gemini 3.7 Flash high 21.1%
  10. 10 Qwen3.7-Plus 9.15%
#ModelScoreSettingDateSource
1 GPT-6 Astra 58.1% max Sep 2026 Official leaderboard
2 Claude Opus 5 44.2% max Jul 2026 Official leaderboard
3 GPT-5.6 Sol 42.1% max Jul 2026 Official leaderboard
4 Grok 4.6 36.7% xhigh Aug 2026 Official leaderboard
5 Qwen3.8 Max (0902) 34.8% xhigh Sep 2026 Official leaderboard
6 Kimi K3 31.4% max Jul 2026 Official leaderboard
7 Claude Opus 4.8 31.1% max May 2026 Official leaderboard
8 GLM-5.3 24.4% max Aug 2026 Official leaderboard
9 Gemini 3.7 Flash 21.1% high Aug 2026 Official leaderboard
10 Qwen3.7-Plus 9.15% Jun 2026 Official leaderboard

Labs report scores for 3 models in their cards. We plot every source in the charts below.

The best score, 58.1%, costs $116 a task

Runs on the cost frontier score higher than any run that's cheaper. The costs are the leaderboard's own estimates, so compare them only within this benchmark.

Qwen OpenAI xAI Cost frontier One model at each effort level
0% 20% 40% 60% $1.00 $10 $100 $1,000 Cost per task, log scale GPT-6 Astra, max GPT-5.6 Sol, max Grok 4.6, xhigh Qwen3.7-Plus 0% 20% 40% 60% $1.00 $10 $100 $1,000 Cost per task, log scale GPT-6 Astra, max GPT-5.6 Sol, max
ModelEffortHarnessScoreCost per taskOutput tokens
GPT-6 Astra max 58.1% $116
Claude Opus 5 max 44.2% $117
GPT-5.6 Sol max 42.1% $25
Grok 4.6 xhigh 36.7% $13
Qwen3.8 Max (0902) xhigh 34.8% $39
Kimi K3 max 31.4% $63
Claude Opus 4.8 max 31.1% $69
GLM-5.3 max 24.4% $71
Gemini 3.7 Flash high 21.1%
Qwen3.7-Plus 9.15% $1.23

2 models have held number one since its release in Sep 2026

Lab claims count from the day the card came out, independent runs from the model’s release. We draw one row each for the benchmark’s own leaderboard and what labs claim in their cards.

Anthropic OpenAI Qwen
Own leaderboard Not tested since Sep 2026
GPT-6 Astra 58.1%
Lab claims Not tested since Sep 2026
Qwen 3.8 Max 51.7%

Scores over time

All the scores we have, one dot each, so a model tested at several settings appears more than once.

On the benchmark’s own leaderboard Reported by the lab in its model card Best so far
0% 20% 40% 60% 80% 100% 2026 Near the ceiling (85%) GPT-6 Astra Qwen 3.8 Max Claude Opus 5 Claude Opus 4.8 0% 20% 40% 60% 80% 100% 2026 Near the ceiling (85%) GPT-6 Astra Qwen 3.8 Max

One lab put it in a launch post in the last six months

Launch posts from every lab that has listed it, one tick per post. Tall colored ticks include it, and short gray ones leave it out.

Qwen
DateLaunch postScoreAs the post writes it
26 Aug 2026 Qwen3.8 Flash 49.9 Application recreation RecreationBench
14 Aug 2026 Qwen 3.8 27B 47.1 Application recreation RecreationBench
3 Aug 2026 Qwen 3.8 Max 51.7 RecreationBench

More in Computer and browser use

All 56 in Computer and browser use
  • BrowseComp In use Hard-to-find facts that need persistent web browsing. Short verifiable answers graded for exact correctness. Claude Fable 5 93.3%
  • ScreenSpot-Pro In use GUI grounding on high-resolution professional-software screenshots. The model must locate the target UI element. GPT-6 Astra 92.7%
  • AndroidWorld In use Agents complete 116 tasks across 20 real Android apps in an emulator. Each task checks the device's system state to judge success. GPT-6 Astra 100%
  • OSWorld 2.1 In use Agents complete 108 long-horizon workflows across 31 self-hosted websites and desktop apps, averaging 318 tool calls. Scored by binary completion and checkpoint partial credit. Claude Opus 5.5 81.8%
  • WebArena-Verified In use Agents complete 812 audited WebArena tasks on self-hosted websites. Deterministic evaluators check the agent's answer and captured network traces. Muse Spark 1.1 69%
  • MobileWorld In use Agents complete 201 tasks across 20 mobile apps, including tasks that need user clarification or MCP tools. Backend checks verify success. Kimi K3 74.4%