Benchmark Almanac

HW-Bench

GPT-6 Astra 67% leads since November 2026
Not in model cards

Agents complete six hardware engineering tasks for a 2.45 GHz antenna using native design and simulation tools. Code checks and calibrated model judges grade outputs against engineer-written criteria. The headline pass rate measures runs that satisfy every critical criterion, averaged across tasks.

Agents run five times per task in the Atlas harness at high reasoning effort. Each task starts from an engineer reference rather than the previous agent output. Arena prohibits automated extraction without written permission. The board dates its initial results to October 29, 2026.

Made by
Arena Physica
Released
November 2026
Kind
Electronics
Level
Expert
Scored
Graded by a model
Answers
Agent task
Tasks
Kept private
Strongest model tested
GPT-6 Astra #3 on the capability index
Last model added
September 2026
Scores refreshed
November 12th, 2026
15
Models scored on its own leaderboard

GPT-6 Astra leads with 67%

Best score per model from the benchmark's own leaderboard, read on November 12th, 2026. Where a model ran with several settings or harnesses, we keep its best. Every run is in the next chart.

15 models. Scroll sideways for the rest.

  1. 67.0 GPT-6 Astra
  2. 57.0 GPT-6.1 Sol
  3. 37.0 GPT-6 Sol
  4. 33.0 Claude Opus 5
  5. 30.0 Claude Opus 4.8
  6. 30.0 GPT-5.6 Sol
  7. 30.0 GPT-6 Luna
  8. 23.0 Claude Fable 5.1
  9. 20.0 Grok 4.7
  10. 20.0 Claude Opus 5.5
  11. 20.0 Claude Sonnet 5.5
  12. 17.0 GLM-5.3
  13. 13.0 Grok 4.6
  14. 7.00 Kimi K3
  15. 3.00 Grok 4.5
  1. 1 GPT-6 Astra 67%
  2. 2 GPT-6.1 Sol 57%
  3. 3 GPT-6 Sol 37%
  4. 4 Claude Opus 5 33%
  5. 5 Claude Opus 4.8 30%
  6. 6 GPT-5.6 Sol 30%
  7. 7 GPT-6 Luna 30%
  8. 8 Claude Fable 5.1 23%
  9. 9 Grok 4.7 20%
  10. 10 Claude Opus 5.5 20%
  11. 11 Claude Sonnet 5.5 20%
  12. 12 GLM-5.3 17%
  13. 13 Grok 4.6 13%
  14. 14 Kimi K3 7.00%
  15. 15 Grok 4.5 3.00%

Source: Official HW-Bench leaderboard

#ModelScoreSettingDateSource
1 GPT-6 Astra 67% Sep 2026 Official leaderboard
2 GPT-6.1 Sol 57% Sep 2026 Official leaderboard
3 GPT-6 Sol 37% Sep 2026 Official leaderboard
4 Claude Opus 5 33% Jul 2026 Official leaderboard
5 Claude Opus 4.8 30% May 2026 Official leaderboard
6 GPT-5.6 Sol 30% Jul 2026 Official leaderboard
7 GPT-6 Luna 30% Sep 2026 Official leaderboard
8 Claude Fable 5.1 23% Sep 2026 Official leaderboard
9 Grok 4.7 20% Sep 2026 Official leaderboard
10 Claude Opus 5.5 20% Sep 2026 Official leaderboard
11 Claude Sonnet 5.5 20% Sep 2026 Official leaderboard
12 GLM-5.3 17% Aug 2026 Official leaderboard
13 Grok 4.6 13% Aug 2026 Official leaderboard
14 Kimi K3 7.00% Jul 2026 Official leaderboard
15 Grok 4.5 3.00% Jul 2026 Official leaderboard

Accuracy vs cost

The line joins the runs that no other run beats on both measures. We redrew this chart from the numbers on the official leaderboard.

40% 60% 80% 100% $0$20$40$60 Cost per task run ($), lower is better Accuracy Claude Opus 4.8: Accuracy 67%, cost per task run $22.4 Grok 4.5: Accuracy 50%, cost per task run $6.53 GPT-5.6 Sol: Accuracy 73%, cost per task run $15.1 Kimi K3: Accuracy 40%, cost per task run $26.2 Claude Opus 5: Accuracy 71%, cost per task run $28.9 Grok 4.6: Accuracy 61%, cost per task run $15.4 GLM-5.3: Accuracy 59%, cost per task run $18.2 Claude Fable 5.1: Accuracy 68%, cost per task run $33.8 Grok 4.7: Accuracy 60%, cost per task run $17.6 Claude Opus 5.5: Accuracy 71%, cost per task run $16.7 GPT-6 Astra: Accuracy 81%, cost per task run $50.5 GPT-6 Luna: Accuracy 59%, cost per task run $0.55 GPT-6 Sol: Accuracy 71%, cost per task run $8.07 Claude Sonnet 5.5: Accuracy 66%, cost per task run $5.75 GPT-6.1 Sol: Accuracy 78%, cost per task run $8.42 GPT-6 AstraGPT-6.1 SolGPT-6 SolClaude Sonnet 5.5GPT-6 Luna 40% 60% 80% 100% $0$20$40$60 Cost per task run ($), lower is better Accuracy Claude Opus 4.8: Accuracy 67%, cost per task run $22.4 Grok 4.5: Accuracy 50%, cost per task run $6.53 GPT-5.6 Sol: Accuracy 73%, cost per task run $15.1 Kimi K3: Accuracy 40%, cost per task run $26.2 Claude Opus 5: Accuracy 71%, cost per task run $28.9 Grok 4.6: Accuracy 61%, cost per task run $15.4 GLM-5.3: Accuracy 59%, cost per task run $18.2 Claude Fable 5.1: Accuracy 68%, cost per task run $33.8 Grok 4.7: Accuracy 60%, cost per task run $17.6 Claude Opus 5.5: Accuracy 71%, cost per task run $16.7 GPT-6 Astra: Accuracy 81%, cost per task run $50.5 GPT-6 Luna: Accuracy 59%, cost per task run $0.55 GPT-6 Sol: Accuracy 71%, cost per task run $8.07 Claude Sonnet 5.5: Accuracy 66%, cost per task run $5.75 GPT-6.1 Sol: Accuracy 78%, cost per task run $8.42 GPT-6 AstraGPT-6.1 SolGPT-6 SolClaude Sonnet 5.5GPT-6 Luna
ModelSettingAccuracyCost per task run
GPT-6 Astra 81% $50.49
GPT-6.1 Sol 78% $8.42
GPT-5.6 Sol 73% $15.05
GPT-6 Sol 71% $8.07
Claude Opus 5.5 71% $16.72
Claude Opus 5 71% $28.87
Claude Fable 5.1 68% $33.84
Claude Opus 4.8 67% $22.36
Claude Sonnet 5.5 66% $5.75
Grok 4.6 61% $15.43
Grok 4.7 60% $17.56
GPT-6 Luna 59% $0.55
GLM-5.3 59% $18.16
Grok 4.5 50% $6.53
Kimi K3 40% $26.18

The best score, 67%, costs $50 a task

Runs on the cost frontier score higher than any run that's cheaper. The costs are the leaderboard's own estimates, so compare them only within this benchmark.

OpenAI Cost frontier One model at each effort level
0% 20% 40% 60% 80% $0.10 $1.00 $10 $100 Cost per task, log scale GPT-6 Astra GPT-6.1 Sol GPT-6 Sol GPT-6 Luna 0% 20% 40% 60% 80% $0.10 $1.00 $10 $100 Cost per task, log scale GPT-6 Astra GPT-6.1 Sol

Source: Official HW-Bench leaderboard

ModelEffortHarnessScoreCost per taskOutput tokens
GPT-6 Astra 67% $50
GPT-6.1 Sol 57% $8.42
GPT-6 Sol 37% $8.07
Claude Opus 5 33% $29
Claude Opus 4.8 30% $22
GPT-5.6 Sol 30% $15
GPT-6 Luna 30% $0.55
Claude Fable 5.1 23% $34
Grok 4.7 20% $18
Claude Opus 5.5 20% $17
Claude Sonnet 5.5 20% $5.75
GLM-5.3 17% $18
Grok 4.6 13% $15
Kimi K3 7.00% $26
Grok 4.5 3.00% $6.53

Number one over time

Lab claims count from the day the card came out, independent runs from the model’s release.

OpenAI
Own leaderboard 1 leader
GPT-6 Astra 67%

Source: Official HW-Bench leaderboard

Scores over time

All the scores we have, one dot each, so a model tested at several settings appears more than once.

On the benchmark’s own leaderboard Best so far
0% 20% 40% 60% 80% 100% 2026 Near the ceiling (85%) GPT-6 Astra Claude Opus 5 Claude Opus 4.8 0% 20% 40% 60% 80% 100% 2026 Near the ceiling (85%) GPT-6 Astra Claude Opus 5

Source: Official HW-Bench leaderboard

No lab model card reports it yet

We read the headline results of 522 launch posts from 52 labs, and none of them list this benchmark.

More in Engineering and CAD

All 23 in Engineering and CAD
  • EEBench In use 13 electrical-engineering circuit and PCB design tasks graded by deterministic build checks, simulation and BOM constraints. No LLM judge. Gemini 4 Carbon 83.9%
  • Parametric CAD Bench In use Agents author editable parametric FreeCAD part designs, graded against held-back references for geometry and spec consistency. Claude Opus 5.5 61%
  • 3DCodeBench In use Agentic procedural 3D modeling via code: author engine-ready 3D assets through software APIs. Graded for executability and shape fidelity. LongCat-2.5 55.4%
  • CADGenBench Only in system cards Agents construct CAD models from design prompts. The generation split is graded for executability and geometric correctness. Claude Fable 5 45.1%
  • PartBench Only in system cards Parametric mechanical part design and CAD-style generation, graded for geometric and manufacturing fidelity. Grok 4.6 58.5%
  • LMBuild Not in model cards Language model agents design 3D structures as parts, joints, materials and assembly steps. Scoring checks that the structures can be built and work as intended. GPT-6 Astra 66.5%