OR-License
Agents turn original client materials into validated operations research solutions without an expert first reformulating the problem.
225 upvotes on Hugging Face papers, the most of any benchmark paper in October 2026.
In the authors’ words
Can large language model (LLM) agents deliver expert-level operations research (OR) solutions directly from original client materials, without prior expert reformulation? Existing OR benchmarks typically begin with self-contained descriptions or predefined optimization tasks, underrepresenting the translation of original business materials into professionally validated solutions. We introduce OR-License, an end-to-end benchmark of 31 instances across 10 industrial scenarios from paid OR engagements totaling USD 3.68 million. It preserves original requirements, input data, deliverable formats, and information boundaries after de-identification. To study information acquisition, we introduce ORSim, a controlled client simulator that discloses business facts without supplying OR expertise. Hidden evaluation checks feasibility and compares objective quality against original expert solutions accepted by clients before benchmark construction. Deployment-Adjusted Quality (DAQ) averages expert-relative quality across instances, assigning zero to infeasible deliveries. Across 11 frontier LLMs using coding-agent harnesses, several exceed the expert reference in conditional objective quality, but none matches its DAQ; the strongest primary Claude Code configuration achieves 88.39% strict feasibility and a DAQ of 0.87 against the expert reference of 1. Client information acquisition extends beyond clarification into solution development and validation, with some later stages showing higher confirmed issue-resolution proportions. Trajectory analysis identifies formulation and implementation failures, while task-equalized research-chain completion rates range from 24% to 45% across models on evaluable tasks. These findings motivate sustained information-gap resolution, faithful problem formulation, and evidence-linked self-improvement for reliable long-horizon agents.
- Level
- Expert Specialists with years of training, often over hours of work
- Scored
- Graded by people Human experts grade or compare the outputs.
- Answers
- Agent task Many steps in a terminal, browser, desktop, app, game or robot.
Grok 4.6 leads with 88.4% among models released by Sep 2026
Best score per entry from the benchmark's own leaderboard, read on 2 Nov 2026. Where a model ran with several settings or harnesses, we keep its best. Names in gray aren't in our model list: fine-tunes, agent systems, products and older models. They don't count toward leaders.
The newest model with a score here came out in Sep 2026, and nobody has tested a newer one. Among the 20 highest-ranked models on the capability index, these have no score yet:
- Claude Opus 5.5
- GPT-6.1 Terra
- GPT-6.1 Sol
- Claude Sonnet 5.5
- and 14 more
- 1 Grok 4.6 high 88.4%
- 2 GPT-5.6 Sol xhigh 84.5%
- 3
Gemini 3.8 Flash high 82.6%
- 4
Qwen 3.8 Max xhigh 78.7%
- 5 Claude Opus 5 xhigh 77.4%
- 6
Muse Spark 1.3 high 74.2%
- 7
DeepSeek V4.1 Flash high 69.7%
- 8
Tencent Hy4 preview high 66.5%
- 9 Doubao-Seed-2.1-Pro high 65.2%
- 10 GLM-5.3-Flash max 62.6%
- 11
Tencent Hy3 high 45.2%
| # | Entry | Score | Setting | Date | Source |
|---|---|---|---|---|---|
| 1 | Grok 4.6 | 88.4% | high | Aug 2026 | Official leaderboard |
| 2 | GPT-5.6 Sol | 84.5% | xhigh | Jul 2026 | Official leaderboard |
| 3 | | 82.6% | high | Sep 2026 | Official leaderboard |
| 4 | | 78.7% | xhigh | Aug 2026 | Official leaderboard |
| 5 | Claude Opus 5 | 77.4% | xhigh | Jul 2026 | Official leaderboard |
| 6 | | 74.2% | high | Sep 2026 | Official leaderboard |
| 7 | | 69.7% | high | Sep 2026 | Official leaderboard |
| 8 | | 66.5% | high | Aug 2026 | Official leaderboard |
| 9 | Doubao-Seed-2.1-Pro | 65.2% | high | Official leaderboard | |
| 10 | GLM-5.3-Flash | 62.6% | max | Aug 2026 | Official leaderboard |
| 11 | | 45.2% | high | Jul 2026 | Official leaderboard |
Number one over time
Lab claims count from the day the card came out, independent runs from the model’s release.
The best score passed 85% in Aug 2026
All the scores we have, one dot each, so a model tested at several settings appears more than once.
No lab model card reports it yet
We read the headline results of 479 launch posts from 51 labs, and none of them list this benchmark.
More in Economic work
- JobBench In use 130 tasks from 35 occupations workers most want delegated. Agent works through messy reference files, graded by fact-anchored binary rubrics. GLM-5.3 61.4%
- GDPval In use Real work tasks from 44 occupations with reference files. Experts blindly compare model deliverables with expert work. Metric is win-or-tie rate.
Seed2.1 Pro 87.9%
- GDPval-AA v2.1 In use Agents with shell and web access complete 220 GDPval gold tasks from 44 occupations. Blind pairwise comparisons give Elo ratings. Claude Opus 5.5 1,846
- AA-Briefcase v1.1 In use Version 1.1 of Artificial Analysis's multi-hour office work evaluation, with Elo ratings fitted by Crowd-BT. Claude Opus 5.5 1,822
- CoWorkBench In use Agent completes long-horizon productivity tasks across computer science, finance, law and medicine. Qwen in-house scoring (method unpublished).
Qwen3.8-Omni-Flash 75.3%
- OfficeQA In use Grounded reasoning over a large corpus of U.S. Treasury Bulletin documents: locate tables and compute precise numeric answers. Claude Opus 4.7 86.3%