BenchCAD
Programmatic CAD: 17,900 execution-verified CadQuery programs over 106 industrial part families. Four matched tasks scored on geometry.
BenchCAD 2.0 (agentic) is in preparation at github.com/BenchCAD-org/benchcad-2.
In the authors’ words
Industrial Computer-Aided Design (CAD) code generation requires models to produce executable parametric programs from visual or textual inputs. Beyond recognizing the outer shape of a part, this task involves understanding its 3D structure, inferring engineering parameters, and choosing CAD operations that reflect how the part would be designed and manufactured. Despite the promise of Multimodal large language models (MLLMs) for this task, they are rarely evaluated on whether these capabilities jointly hold in realistic industrial CAD settings. We present BenchCAD, a unified benchmark for industrial CAD reasoning. BenchCAD contains 17,900 execution-verified CadQuery programs across 106 industrial part families, including bevel gears, compression springs, twist drills, and other reusable engineering designs. It evaluates models through visual question answering, code question answering, image-to-code generation, and instruction-guided code editing, enabling fine-grained analysis across perception, parametric abstraction, and executable program synthesis. Across 10+ frontier models, BenchCAD shows that current systems often recover coarse outer geometry but fail to produce faithful parametric CAD programs. Common failures include missing fine 3D structure, misinterpreting industrial design parameters, and replacing essential operations such as sweeps, lofts, and twist-extrudes with simpler sketch-and-extrude patterns. Fine-tuning and reinforcement learning improve in-distribution performance, but generalization to unseen part families remains limited. These results position BenchCAD as a benchmark for measuring and improving the industrial readiness of multimodal CAD automation.
- Level
- University Undergraduates and entry-level professionals
- Scored
- Checked automatically A program checks the result against an answer key, unit tests or the expected final state.
- Answers
- Code A program or patch run against tests.
GPT-6 Astra leads with 95.9%
Best score per model as labs report them in their own cards. Labs pick their own settings, so these compare less cleanly than independent runs.
- 1 GPT-6 Astra 95.9%
- 2 Kimi K3.1 Pro 92.7%
- 3 GPT-5.6 Terra 78.2%
- 4 Claude Sonnet 5.5 74.7%
- 5 GPT-5.6 Luna 73.9%
- 6 Claude Opus 5.5 73%
- 7 Claude Haiku 5.5 67%
- 8 Claude Fable 5.1 43.7%
- 9 Claude Opus 5 36.6%
- 10 Claude Sonnet 5 26.6%
| # | Model | Score | Setting | Date | Source |
|---|---|---|---|---|---|
| 1 | GPT-6 Astra | 95.9% | Sep 2026 | Model card | |
| 2 | Kimi K3.1 Pro | 92.7% | Oct 2026 | Model card | |
| 3 | GPT-5.6 Terra | 78.2% | Jul 2026 | Model card | |
| 4 | Claude Sonnet 5.5 | 74.7% | Sep 2026 | Model card | |
| 5 | GPT-5.6 Luna | 73.9% | Jul 2026 | Model card | |
| 6 | Claude Opus 5.5 | 73% | Sep 2026 | Model card | |
| 7 | Claude Haiku 5.5 | 67% | Oct 2026 | Model card | |
| 8 | Claude Fable 5.1 | 43.7% | Sep 2026 | Model card | |
| 9 | Claude Opus 5 | 36.6% | Jul 2026 | Model card | |
| 10 | Claude Sonnet 5 | 26.6% | Jun 2026 | Model card |
3 models have held number one since its release in May 2026
Lab claims count from the day the card came out, independent runs from the model’s release.
The best score passed 85% in Sep 2026
All the scores we have, one dot each, so a model tested at several settings appears more than once.
2 labs put it in a launch post in the last six months
Launch posts from every lab that has listed it, one tick per post. Tall colored ticks include it, and short gray ones leave it out.
| Date | Launch post | Score | As the post writes it |
|---|---|---|---|
| 24 Oct 2026 | Kimi K3.1 Pro | 92.7 | BenchCAD w/ tools (w/o tools 72.8) |
| 3 Sep 2026 | GPT-6 Astra | 95.9 | BenchCAD |
| 9 Jul 2026 | GPT-5.6 Luna | 73.9 | BenchCAD (python tool) 73.9 (no tools 63.1) |
| 9 Jul 2026 | GPT-5.6 Terra | 78.2 | BenchCAD (python tool) 78.2 (no tools 62.3) |
6 system cards report it, from 1 lab
Besides launch posts, we read the system cards, model cards and technical reports labs publish with a release. They report far more, including safety and frontier-risk tests.
| Date | Document | Score | As the document writes it | Section |
|---|---|---|---|---|
| 7 Oct 2026 | Claude Haiku 5.5 system card | 67 | BenchCAD Vision2Code, 1,000-file subset, voxel IoU 0.670 without tools (0.870 with tools), max effort (8.9.2) | Capabilities |
| 28 Sep 2026 | Claude Sonnet 5.5 system card | 74.7 | BenchCAD Vision2Code, 1,000-file subset, voxel IoU 0.747 without tools (0.963 with tools) (8.13.2, Figure 8.13.2.A) | Capabilities |
| 22 Sep 2026 | Claude Opus 5.5 system card | 73 | BenchCAD Vision2Code, 1,000-file subset, voxel IoU 0.730 without tools (0.962 with tools), max effort, 256 px views (8.13.2, Figure 8.13.2.A) | Capabilities |
| 1 Sep 2026 | Claude Fable 5.1 system card | 43.7 | BenchCAD Vision2Code, 1,000-file subset, voxel IoU 0.437 without tools; 0.843 with tools (8.14.2) | Capabilities |
| 24 Jul 2026 | Claude Opus 5 system card | 36.6 | BenchCAD Vision2Code, 1,000-file subset, voxel IoU 0.366 without tools (0.821 with tools), max effort, five runs (8.12.2) | Capabilities |
| 30 Jun 2026 | Claude Sonnet 5 system card | 26.6 | BenchCAD Vision2Code, 1,000-file subset, voxel IoU 0.266 without tools (0.373 with tools), max effort (8.10.3) | Capabilities |
More in Engineering and CAD
- EEBench In use 13 electrical-engineering circuit and PCB design tasks graded by deterministic build checks, simulation and BOM constraints. No LLM judge. Claude Opus 5.5 75%
- Parametric CAD Bench In use Agents author editable parametric FreeCAD part designs, graded against held-back references for geometry and spec consistency. Claude Opus 5.5 61%
- CADGenBench Only in system cards Agents construct CAD models from design prompts. The generation split is graded for executability and geometric correctness. Claude Fable 5 45.1%
- 3DCodeBench Only in system cards Agentic procedural 3D modeling via code: author engine-ready 3D assets through software APIs. Graded for executability and shape fidelity. Grok 4.6 54%
- PartBench Only in system cards Parametric mechanical part design and CAD-style generation, graded for geometric and manufacturing fidelity. Grok 4.6 58.5%
- Physical-Engineering's Last Exam Not in model cards Agents produce billable engineering deliverables such as calculation packages, drawings and design reviews in 85 tasks across electrical, civil, structural and mechanical engineering. Automatic graders check constraints, safety limits and the design objective. Claude Opus 5.5 0.49