Physical-Engineering's Last Exam
Agents produce billable engineering deliverables such as calculation packages, drawings and design reviews in 85 tasks across electrical, civil, structural and mechanical engineering. Automatic graders check constraints, safety limits and the design objective.
29 public dev tasks and 56 hidden test tasks. Claude Opus 5.5 in Claude Code leads the board at 0.483.
In the authors’ words
Frontier language models now score above 90% on popular knowledge benchmarks and are increasingly deployed as agents that carry out multi-hour professional work. Yet no existing benchmark tests whether they can perform the real-world, economically valuable work of a professional physical engineer, such as modelling physics and legal code requirements across a multi-format project corpus, optimising material and installation costs, and issuing 2D/3D engineering design packages that a licensed reviewer would sign off on. Existing engineering benchmarks focus on textbook question answering, isolated tool calls or self-contained toy design problems. As a result, most of these benchmarks are already saturated by leading agents and solving them is far away from delivering real engineering design. We introduce PLE, a benchmark of 85 long-horizon, agentic tasks in professional physical engineering consisting of electrical, civil, structural and mechanical engineering, where solving a task means completing a realistic, billable engineering deliverable. PLE combines two complementary curation paths. Expert-Authored tasks are created end to end by practising engineers from a professional data provider's expert pool. Co-Authored tasks start from problem statements written by practising engineers; an adversarial Creator-Solver agent pipeline constructs each task's realistic project environment, executable grader and reference deliverables before sending them to expert engineers for validation and difficulty intensification. We evaluate a variety of frontier LLMs under seven publicly available agent harnesses. Even the strongest configuration reaches a mean score of only 50.6%, while spending 27M tokens and 63 minutes per task on average, demonstrating the substantial gap between current agents and autonomous physical engineering design.
- Level
- Expert Specialists with years of training, often over hours of work
- Scored
- Checked automatically A program checks the result against an answer key, unit tests or the expected final state.
- Answers
- Agent task Many steps in a terminal, browser, desktop, app, game or robot.
Claude Opus 5.5 leads with 0.49
Best score per model from the benchmark's own leaderboard, read on 2 Nov 2026. Where a model ran with several settings or harnesses, we keep its best. Every run is in the next chart.
- 0.49 Claude Opus 5.5
- 0.46 GPT-6 Astra
- 0.45 Claude Sonnet 5.5
- 0.43 Claude Opus 5
- 0.41
Gemini 4 Argon
- 0.34 Claude Fable 5
- 0.30 Grok 4.7
- 0.28
Gemini 3.8 Flash
- 0.25 GLM-5.2
- 0.23
DeepSeek V4.1 Flash
- 0.15
Gemini 3.1 Pro
- 0.06
Qwen3-Coder-480B-A35B
- 0.06 Kimi K2 Thinking
- 0.06
MiniMax-M2
- 0.05 gpt-oss-120b
- 0.04
Gemma 4 26B A4B
- 0.04
Qwen3-Next-80B-A3B
- 0.01
Llama 4 Maverick
- 1 Claude Opus 5.5 0.49
- 2 GPT-6 Astra 0.46
- 3 Claude Sonnet 5.5 0.45
- 4 Claude Opus 5 0.43
- 5
Gemini 4 Argon 0.41
- 6 Claude Fable 5 0.34
- 7 Grok 4.7 0.30
- 8
Gemini 3.8 Flash 0.28
- 9 GLM-5.2 0.25
- 10
DeepSeek V4.1 Flash 0.23
- 11
Gemini 3.1 Pro 0.15
- 12
Qwen3-Coder-480B-A35B 0.06
- 13 Kimi K2 Thinking 0.06
- 14
MiniMax-M2 0.06
- 15 gpt-oss-120b 0.05
- 16
Gemma 4 26B A4B 0.04
- 17
Qwen3-Next-80B-A3B 0.04
- 18
Llama 4 Maverick 0.01
| # | Model | Score | Setting | Date | Source |
|---|---|---|---|---|---|
| 1 | Claude Opus 5.5 | 0.49 | openhands | Sep 2026 | Official leaderboard |
| 2 | GPT-6 Astra | 0.46 | codex | Sep 2026 | Official leaderboard |
| 3 | Claude Sonnet 5.5 | 0.45 | hermes | Sep 2026 | Official leaderboard |
| 4 | Claude Opus 5 | 0.43 | openhands | Jul 2026 | Official leaderboard |
| 5 | | 0.41 | claude-code | Sep 2026 | Official leaderboard |
| 6 | Claude Fable 5 | 0.34 | claude-code | Jun 2026 | Official leaderboard |
| 7 | Grok 4.7 | 0.30 | pi | Sep 2026 | Official leaderboard |
| 8 | | 0.28 | openhands | Sep 2026 | Official leaderboard |
| 9 | GLM-5.2 | 0.25 | pi | Jun 2026 | Official leaderboard |
| 10 | | 0.23 | pi | Sep 2026 | Official leaderboard |
| 11 | | 0.15 | hermes | Feb 2026 | Official leaderboard |
| 12 | | 0.06 | claude-code | Jul 2025 | Official leaderboard |
| 13 | Kimi K2 Thinking | 0.06 | opencode | Nov 2025 | Official leaderboard |
| 14 | | 0.06 | claude-code | Oct 2025 | Official leaderboard |
| 15 | gpt-oss-120b | 0.05 | openhands | Aug 2025 | Official leaderboard |
| 16 | | 0.04 | openhands | Apr 2026 | Official leaderboard |
| 17 | | 0.04 | openhands | Sep 2025 | Official leaderboard |
| 18 | | 0.01 | pi | Apr 2025 | Official leaderboard |
The best score, 0.49, costs $15 a task
Runs on the cost frontier score higher than any run that's cheaper. The costs are the leaderboard's own estimates, so compare them only within this benchmark.
| Model | Effort | Harness | Score | Cost per task | Output tokens |
|---|---|---|---|---|---|
| Claude Opus 5.5 | openhands | 0.49 | $15 | ||
| Claude Opus 5.5 | claude-code | 0.48 | $17 | ||
| Claude Opus 5.5 | agy | 0.48 | $42 | ||
| Claude Opus 5.5 | pi | 0.47 | $13 | ||
| Claude Opus 5.5 | hermes | 0.47 | $16 | ||
| Claude Opus 5.5 | opencode | 0.47 | $13 | ||
| Claude Opus 5.5 | codex | 0.46 | $38 | ||
| GPT-6 Astra | codex | 0.46 | $9.53 | ||
| Claude Sonnet 5.5 | hermes | 0.45 | $15 | ||
| Claude Sonnet 5.5 | agy | 0.45 | $34 | ||
| Claude Sonnet 5.5 | pi | 0.44 | $12 | ||
| Claude Sonnet 5.5 | claude-code | 0.43 | $19 | ||
| Claude Sonnet 5.5 | codex | 0.43 | $43 | ||
| Claude Opus 5 | openhands | 0.43 | $17 | ||
| Claude Sonnet 5.5 | opencode | 0.42 | $11 | ||
| Claude Opus 5 | pi | 0.42 | $15 | ||
| Claude Opus 5 | agy | 0.41 | $30 | ||
| | claude-code | 0.41 | $8.51 | ||
| Claude Sonnet 5.5 | openhands | 0.41 | $16 | ||
| | agy | 0.41 | $7.70 | ||
| Claude Opus 5 | opencode | 0.40 | $13 | ||
| Claude Opus 5 | hermes | 0.40 | $14 | ||
| Claude Opus 5 | claude-code | 0.40 | $17 | ||
| Claude Opus 5 | codex | 0.39 | $21 | ||
| | codex | 0.39 | $7.13 | ||
| | hermes | 0.39 | $7.92 | ||
| | opencode | 0.37 | $7.04 | ||
| | pi | 0.37 | $7.00 | ||
| Claude Fable 5 | claude-code | 0.34 | $26 | ||
| Claude Fable 5 | codex | 0.34 | $29 | ||
| Claude Fable 5 | pi | 0.34 | $21 | ||
| Claude Fable 5 | opencode | 0.34 | $23 | ||
| Claude Fable 5 | hermes | 0.33 | $24 | ||
| Claude Fable 5 | openhands | 0.33 | $26 | ||
| Claude Fable 5 | agy | 0.33 | $54 | ||
| Grok 4.7 | pi | 0.30 | $8.01 | ||
| Grok 4.7 | grok-build | 0.30 | $10 | ||
| | openhands | 0.29 | $7.54 | ||
| Grok 4.7 | opencode | 0.28 | $10 | ||
| | openhands | 0.28 | $3.21 | ||
| | codex | 0.28 | $3.61 | ||
| Grok 4.7 | agy | 0.28 | $17 | ||
| | agy | 0.28 | $2.60 | ||
| Grok 4.7 | openhands | 0.28 | $8.89 | ||
| | claude-code | 0.27 | $2.61 | ||
| | pi | 0.27 | $2.68 | ||
| Grok 4.7 | claude-code | 0.27 | $13 | ||
| | opencode | 0.26 | $2.45 | ||
| Grok 4.7 | codex | 0.26 | $7.02 | ||
| Grok 4.7 | hermes | 0.26 | $10 | ||
| GLM-5.2 | pi | 0.25 | $14 | ||
| GLM-5.2 | codex | 0.25 | $13 | ||
| GLM-5.2 | opencode | 0.24 | |||
| GLM-5.2 | hermes | 0.24 | $11 | ||
| GLM-5.2 | agy | 0.24 | $13 | ||
| GLM-5.2 | openhands | 0.23 | $16 | ||
| | pi | 0.23 | $4.54 | ||
| | opencode | 0.23 | $3.46 | ||
| GLM-5.2 | claude-code | 0.22 | $15 | ||
| | hermes | 0.17 | $2.39 | ||
| | hermes | 0.15 | $2.09 | ||
| | openhands | 0.15 | $4.67 | ||
| | codex | 0.14 | $2.80 | ||
| | claude-code | 0.14 | $3.65 | ||
| | dsh | 0.14 | $6.30 | ||
| | pi | 0.14 | $2.76 | ||
| | opencode | 0.14 | $2.67 | ||
| | agy | 0.14 | $2.47 | ||
| | claude-code | 0.12 | $3.74 | ||
| | agy | 0.09 | $3.50 | ||
| | openhands | 0.06 | $1.65 | ||
| | claude-code | 0.06 | $0.13 | ||
| | hermes | 0.06 | $0.12 | ||
| Kimi K2 Thinking | opencode | 0.06 | $0.28 | ||
| | openhands | 0.06 | $0.21 | ||
| | claude-code | 0.06 | $0.10 | ||
| | hermes | 0.06 | $0.10 | ||
| | openhands | 0.05 | $0.13 | ||
| | hermes | 0.05 | $2.98 | ||
| Kimi K2 Thinking | hermes | 0.05 | $0.28 | ||
| | codex | 0.05 | $0.11 | ||
| gpt-oss-120b | openhands | 0.05 | $0.14 | ||
| | pi | 0.05 | $0.09 | ||
| | codex | 0.05 | $0.09 | ||
| Kimi K2 Thinking | claude-code | 0.05 | $0.25 | ||
| | opencode | 0.04 | $0.25 | ||
| | openhands | 0.04 | $0.61 | ||
| | agy | 0.04 | $0.49 | ||
| Kimi K2 Thinking | openhands | 0.04 | $0.25 | ||
| | openhands | 0.04 | $0.77 | ||
| | pi | 0.04 | $0.15 | ||
| gpt-oss-120b | claude-code | 0.04 | $0.11 | ||
| gpt-oss-120b | agy | 0.04 | |||
| | opencode | 0.04 | $0.08 | ||
| | agy | 0.04 | $0.21 | ||
| | codex | 0.04 | $2.08 | ||
| gpt-oss-120b | pi | 0.04 | $0.12 | ||
| Kimi K2 Thinking | pi | 0.04 | $0.20 | ||
| gpt-oss-120b | codex | 0.04 | $0.18 | ||
| | pi | 0.04 | $0.29 | ||
| | claude-code | 0.04 | $0.53 | ||
| gpt-oss-120b | opencode | 0.04 | $0.16 | ||
| gpt-oss-120b | hermes | 0.04 | $0.16 | ||
| | agy | 0.04 | $0.33 | ||
| Kimi K2 Thinking | agy | 0.03 | $0.29 | ||
| Kimi K2 Thinking | codex | 0.03 | $0.26 | ||
| | pi | 0.03 | $1.05 | ||
| | opencode | 0.03 | $0.45 | ||
| | opencode | 0.03 | $0.54 | ||
| | claude-code | 0.03 | $0.54 | ||
| | hermes | 0.02 | $0.41 | ||
| | hermes | 0.02 | $0.29 | ||
| | agy | 0.01 | $0.40 | ||
| | codex | 0.01 | $0.56 | ||
| | pi | 0.01 | $0.34 | ||
| | openhands | 0.01 | $0.18 | ||
| | codex | 0.01 | $0.20 | ||
| | claude-code | 0.00 | $0.06 | ||
| | agy | 0.00 | $0.06 | ||
| | opencode | 0.00 | $0.004 | ||
| | codex | 0.00 | $0.10 | ||
| | hermes | 0.00 | $0.05 |
Number one over time
Lab claims count from the day the card came out, independent runs from the model’s release.
Scores over time
All the scores we have, one dot each, so a model tested at several settings appears more than once.
No lab model card reports it yet
We read the headline results of 479 launch posts from 51 labs, and none of them list this benchmark.
More in Engineering and CAD
- BenchCAD In use Programmatic CAD: 17,900 execution-verified CadQuery programs over 106 industrial part families. Four matched tasks scored on geometry. GPT-6 Astra 95.9%
- EEBench In use 13 electrical-engineering circuit and PCB design tasks graded by deterministic build checks, simulation and BOM constraints. No LLM judge. Claude Opus 5.5 75%
- Parametric CAD Bench In use Agents author editable parametric FreeCAD part designs, graded against held-back references for geometry and spec consistency. Claude Opus 5.5 61%
- CADGenBench Only in system cards Agents construct CAD models from design prompts. The generation split is graded for executability and geometric correctness. Claude Fable 5 45.1%
- 3DCodeBench Only in system cards Agentic procedural 3D modeling via code: author engine-ready 3D assets through software APIs. Graded for executability and shape fidelity. Grok 4.6 54%
- PartBench Only in system cards Parametric mechanical part design and CAD-style generation, graded for geometric and manufacturing fidelity. Grok 4.6 58.5%