Benchmark Almanac

Physical-Engineering's Last Exam

Claude Opus 5.5 0.49 leads since Oct 2026 Not in model cards

Agents produce billable engineering deliverables such as calculation packages, drawings and design reviews in 85 tasks across electrical, civil, structural and mechanical engineering. Automatic graders check constraints, safety limits and the design objective.

29 public dev tasks and 56 hidden test tasks. Claude Opus 5.5 in Claude Code leads the board at 0.483.

In the authors’ words

Frontier language models now score above 90% on popular knowledge benchmarks and are increasingly deployed as agents that carry out multi-hour professional work. Yet no existing benchmark tests whether they can perform the real-world, economically valuable work of a professional physical engineer, such as modelling physics and legal code requirements across a multi-format project corpus, optimising material and installation costs, and issuing 2D/3D engineering design packages that a licensed reviewer would sign off on. Existing engineering benchmarks focus on textbook question answering, isolated tool calls or self-contained toy design problems. As a result, most of these benchmarks are already saturated by leading agents and solving them is far away from delivering real engineering design. We introduce PLE, a benchmark of 85 long-horizon, agentic tasks in professional physical engineering consisting of electrical, civil, structural and mechanical engineering, where solving a task means completing a realistic, billable engineering deliverable. PLE combines two complementary curation paths. Expert-Authored tasks are created end to end by practising engineers from a professional data provider's expert pool. Co-Authored tasks start from problem statements written by practising engineers; an adversarial Creator-Solver agent pipeline constructs each task's realistic project environment, executable grader and reference deliverables before sending them to expert engineers for validation and difficulty intensification. We evaluate a variety of frontier LLMs under seven publicly available agent harnesses. Even the strongest configuration reaches a mean score of only 50.6%, while spending 27M tokens and 63 minutes per task on average, demonstrating the substantial gap between current agents and autonomous physical engineering design.

Abstract of PLE: Physical-Engineering's Last Exam (arXiv:2610.30537, CC0)
Made by Google Cloud AI Research Released Oct 2026 CAD models Public tasks plus a private test set Last model added Sep 2026 Scores refreshed 2 Nov 2026
Level
Expert Specialists with years of training, often over hours of work
Scored
Checked automatically A program checks the result against an answer key, unit tests or the expected final state.
Answers
Agent task Many steps in a terminal, browser, desktop, app, game or robot.
18
Models scored on its own leaderboard

Claude Opus 5.5 leads with 0.49

Best score per model from the benchmark's own leaderboard, read on 2 Nov 2026. Where a model ran with several settings or harnesses, we keep its best. Every run is in the next chart.

  1. 1 Claude Opus 5.5 0.49
  2. 2 GPT-6 Astra 0.46
  3. 3 Claude Sonnet 5.5 0.45
  4. 4 Claude Opus 5 0.43
  5. 5 Gemini 4 Argon 0.41
  6. 6 Claude Fable 5 0.34
  7. 7 Grok 4.7 0.30
  8. 8 Gemini 3.8 Flash 0.28
  9. 9 GLM-5.2 0.25
  10. 10 DeepSeek V4.1 Flash 0.23
  11. 11 Gemini 3.1 Pro 0.15
  12. 12 Qwen3-Coder-480B-A35B 0.06
  13. 13 Kimi K2 Thinking 0.06
  14. 14 MiniMax-M2 0.06
  15. 15 gpt-oss-120b 0.05
  16. 16 Gemma 4 26B A4B 0.04
  17. 17 Qwen3-Next-80B-A3B 0.04
  18. 18 Llama 4 Maverick 0.01
#ModelScoreSettingDateSource
1 Claude Opus 5.5 0.49 openhands Sep 2026 Official leaderboard
2 GPT-6 Astra 0.46 codex Sep 2026 Official leaderboard
3 Claude Sonnet 5.5 0.45 hermes Sep 2026 Official leaderboard
4 Claude Opus 5 0.43 openhands Jul 2026 Official leaderboard
5 Gemini 4 Argon 0.41 claude-code Sep 2026 Official leaderboard
6 Claude Fable 5 0.34 claude-code Jun 2026 Official leaderboard
7 Grok 4.7 0.30 pi Sep 2026 Official leaderboard
8 Gemini 3.8 Flash 0.28 openhands Sep 2026 Official leaderboard
9 GLM-5.2 0.25 pi Jun 2026 Official leaderboard
10 DeepSeek V4.1 Flash 0.23 pi Sep 2026 Official leaderboard
11 Gemini 3.1 Pro 0.15 hermes Feb 2026 Official leaderboard
12 Qwen3-Coder-480B-A35B 0.06 claude-code Jul 2025 Official leaderboard
13 Kimi K2 Thinking 0.06 opencode Nov 2025 Official leaderboard
14 MiniMax-M2 0.06 claude-code Oct 2025 Official leaderboard
15 gpt-oss-120b 0.05 openhands Aug 2025 Official leaderboard
16 Gemma 4 26B A4B 0.04 openhands Apr 2026 Official leaderboard
17 Qwen3-Next-80B-A3B 0.04 openhands Sep 2025 Official leaderboard
18 Llama 4 Maverick 0.01 pi Apr 2025 Official leaderboard

The best score, 0.49, costs $15 a task

Runs on the cost frontier score higher than any run that's cheaper. The costs are the leaderboard's own estimates, so compare them only within this benchmark.

Meta Qwen MiniMax Google DeepMind OpenAI DeepSeek Anthropic Cost frontier One model at each effort level
0.00 0.10 0.20 0.30 0.40 0.50 $0.001 $0.01 $0.10 $1.00 $10 $100 Cost per task, log scale Claude Opus 5.5 Claude Opus 5.5 Gemini 4 Argon Gemini 4 Argon Gemini 4 Argon Gemini 3.8 Flash Gemini 3.8 Flash 0.00 0.10 0.20 0.30 0.40 0.50 $0.001 $0.01 $0.10 $1.00 $10 $100 Cost per task, log scale Claude Opus 5.5 GPT-6 Astra
ModelEffortHarnessScoreCost per taskOutput tokens
Claude Opus 5.5 openhands 0.49 $15
Claude Opus 5.5 claude-code 0.48 $17
Claude Opus 5.5 agy 0.48 $42
Claude Opus 5.5 pi 0.47 $13
Claude Opus 5.5 hermes 0.47 $16
Claude Opus 5.5 opencode 0.47 $13
Claude Opus 5.5 codex 0.46 $38
GPT-6 Astra codex 0.46 $9.53
Claude Sonnet 5.5 hermes 0.45 $15
Claude Sonnet 5.5 agy 0.45 $34
Claude Sonnet 5.5 pi 0.44 $12
Claude Sonnet 5.5 claude-code 0.43 $19
Claude Sonnet 5.5 codex 0.43 $43
Claude Opus 5 openhands 0.43 $17
Claude Sonnet 5.5 opencode 0.42 $11
Claude Opus 5 pi 0.42 $15
Claude Opus 5 agy 0.41 $30
Gemini 4 Argon claude-code 0.41 $8.51
Claude Sonnet 5.5 openhands 0.41 $16
Gemini 4 Argon agy 0.41 $7.70
Claude Opus 5 opencode 0.40 $13
Claude Opus 5 hermes 0.40 $14
Claude Opus 5 claude-code 0.40 $17
Claude Opus 5 codex 0.39 $21
Gemini 4 Argon codex 0.39 $7.13
Gemini 4 Argon hermes 0.39 $7.92
Gemini 4 Argon opencode 0.37 $7.04
Gemini 4 Argon pi 0.37 $7.00
Claude Fable 5 claude-code 0.34 $26
Claude Fable 5 codex 0.34 $29
Claude Fable 5 pi 0.34 $21
Claude Fable 5 opencode 0.34 $23
Claude Fable 5 hermes 0.33 $24
Claude Fable 5 openhands 0.33 $26
Claude Fable 5 agy 0.33 $54
Grok 4.7 pi 0.30 $8.01
Grok 4.7 grok-build 0.30 $10
Gemini 4 Argon openhands 0.29 $7.54
Grok 4.7 opencode 0.28 $10
Gemini 3.8 Flash openhands 0.28 $3.21
Gemini 3.8 Flash codex 0.28 $3.61
Grok 4.7 agy 0.28 $17
Gemini 3.8 Flash agy 0.28 $2.60
Grok 4.7 openhands 0.28 $8.89
Gemini 3.8 Flash claude-code 0.27 $2.61
Gemini 3.8 Flash pi 0.27 $2.68
Grok 4.7 claude-code 0.27 $13
Gemini 3.8 Flash opencode 0.26 $2.45
Grok 4.7 codex 0.26 $7.02
Grok 4.7 hermes 0.26 $10
GLM-5.2 pi 0.25 $14
GLM-5.2 codex 0.25 $13
GLM-5.2 opencode 0.24
GLM-5.2 hermes 0.24 $11
GLM-5.2 agy 0.24 $13
GLM-5.2 openhands 0.23 $16
DeepSeek V4.1 Flash pi 0.23 $4.54
DeepSeek V4.1 Flash opencode 0.23 $3.46
GLM-5.2 claude-code 0.22 $15
Gemini 3.8 Flash hermes 0.17 $2.39
Gemini 3.1 Pro hermes 0.15 $2.09
Gemini 3.1 Pro openhands 0.15 $4.67
Gemini 3.1 Pro codex 0.14 $2.80
Gemini 3.1 Pro claude-code 0.14 $3.65
DeepSeek V4.1 Flash dsh 0.14 $6.30
Gemini 3.1 Pro pi 0.14 $2.76
Gemini 3.1 Pro opencode 0.14 $2.67
Gemini 3.1 Pro agy 0.14 $2.47
DeepSeek V4.1 Flash claude-code 0.12 $3.74
DeepSeek V4.1 Flash agy 0.09 $3.50
DeepSeek V4.1 Flash openhands 0.06 $1.65
Qwen3-Coder-480B-A35B claude-code 0.06 $0.13
Qwen3-Coder-480B-A35B hermes 0.06 $0.12
Kimi K2 Thinking opencode 0.06 $0.28
Qwen3-Coder-480B-A35B openhands 0.06 $0.21
MiniMax-M2 claude-code 0.06 $0.10
MiniMax-M2 hermes 0.06 $0.10
MiniMax-M2 openhands 0.05 $0.13
DeepSeek V4.1 Flash hermes 0.05 $2.98
Kimi K2 Thinking hermes 0.05 $0.28
Qwen3-Coder-480B-A35B codex 0.05 $0.11
gpt-oss-120b openhands 0.05 $0.14
MiniMax-M2 pi 0.05 $0.09
MiniMax-M2 codex 0.05 $0.09
Kimi K2 Thinking claude-code 0.05 $0.25
Qwen3-Coder-480B-A35B opencode 0.04 $0.25
Gemma 4 26B A4B openhands 0.04 $0.61
Qwen3-Coder-480B-A35B agy 0.04 $0.49
Kimi K2 Thinking openhands 0.04 $0.25
Qwen3-Next-80B-A3B openhands 0.04 $0.77
Qwen3-Coder-480B-A35B pi 0.04 $0.15
gpt-oss-120b claude-code 0.04 $0.11
gpt-oss-120b agy 0.04
MiniMax-M2 opencode 0.04 $0.08
Gemma 4 26B A4B agy 0.04 $0.21
DeepSeek V4.1 Flash codex 0.04 $2.08
gpt-oss-120b pi 0.04 $0.12
Kimi K2 Thinking pi 0.04 $0.20
gpt-oss-120b codex 0.04 $0.18
Gemma 4 26B A4B pi 0.04 $0.29
Gemma 4 26B A4B claude-code 0.04 $0.53
gpt-oss-120b opencode 0.04 $0.16
gpt-oss-120b hermes 0.04 $0.16
MiniMax-M2 agy 0.04 $0.33
Kimi K2 Thinking agy 0.03 $0.29
Kimi K2 Thinking codex 0.03 $0.26
Qwen3-Next-80B-A3B pi 0.03 $1.05
Gemma 4 26B A4B opencode 0.03 $0.45
Qwen3-Next-80B-A3B opencode 0.03 $0.54
Qwen3-Next-80B-A3B claude-code 0.03 $0.54
Qwen3-Next-80B-A3B hermes 0.02 $0.41
Gemma 4 26B A4B hermes 0.02 $0.29
Qwen3-Next-80B-A3B agy 0.01 $0.40
Qwen3-Next-80B-A3B codex 0.01 $0.56
Llama 4 Maverick pi 0.01 $0.34
Llama 4 Maverick openhands 0.01 $0.18
Gemma 4 26B A4B codex 0.01 $0.20
Llama 4 Maverick claude-code 0.00 $0.06
Llama 4 Maverick agy 0.00 $0.06
Llama 4 Maverick opencode 0.00 $0.004
Llama 4 Maverick codex 0.00 $0.10
Llama 4 Maverick hermes 0.00 $0.05

Number one over time

Lab claims count from the day the card came out, independent runs from the model’s release.

Anthropic
Own leaderboard 1 leader

Scores over time

All the scores we have, one dot each, so a model tested at several settings appears more than once.

On the benchmark’s own leaderboard
0.00 0.10 0.20 0.30 0.40 0.50 2025 2026 0.00 0.10 0.20 0.30 0.40 0.50 2025 2026

No lab model card reports it yet

We read the headline results of 479 launch posts from 51 labs, and none of them list this benchmark.

More in Engineering and CAD

All 18 in Engineering and CAD
  • BenchCAD In use Programmatic CAD: 17,900 execution-verified CadQuery programs over 106 industrial part families. Four matched tasks scored on geometry. GPT-6 Astra 95.9%
  • EEBench In use 13 electrical-engineering circuit and PCB design tasks graded by deterministic build checks, simulation and BOM constraints. No LLM judge. Claude Opus 5.5 75%
  • Parametric CAD Bench In use Agents author editable parametric FreeCAD part designs, graded against held-back references for geometry and spec consistency. Claude Opus 5.5 61%
  • CADGenBench Only in system cards Agents construct CAD models from design prompts. The generation split is graded for executability and geometric correctness. Claude Fable 5 45.1%
  • 3DCodeBench Only in system cards Agentic procedural 3D modeling via code: author engine-ready 3D assets through software APIs. Graded for executability and shape fidelity. Grok 4.6 54%
  • PartBench Only in system cards Parametric mechanical part design and CAD-style generation, graded for geometric and manufacturing fidelity. Grok 4.6 58.5%