Benchmark Almanac

OR-License

Grok 4.6 88.4% leads since Oct 2026 Not in model cards

Agents turn original client materials into validated operations research solutions without an expert first reformulating the problem.

225 upvotes on Hugging Face papers, the most of any benchmark paper in October 2026.

In the authors’ words

Can large language model (LLM) agents deliver expert-level operations research (OR) solutions directly from original client materials, without prior expert reformulation? Existing OR benchmarks typically begin with self-contained descriptions or predefined optimization tasks, underrepresenting the translation of original business materials into professionally validated solutions. We introduce OR-License, an end-to-end benchmark of 31 instances across 10 industrial scenarios from paid OR engagements totaling USD 3.68 million. It preserves original requirements, input data, deliverable formats, and information boundaries after de-identification. To study information acquisition, we introduce ORSim, a controlled client simulator that discloses business facts without supplying OR expertise. Hidden evaluation checks feasibility and compares objective quality against original expert solutions accepted by clients before benchmark construction. Deployment-Adjusted Quality (DAQ) averages expert-relative quality across instances, assigning zero to infeasible deliveries. Across 11 frontier LLMs using coding-agent harnesses, several exceed the expert reference in conditional objective quality, but none matches its DAQ; the strongest primary Claude Code configuration achieves 88.39% strict feasibility and a DAQ of 0.87 against the expert reference of 1. Client information acquisition extends beyond clarification into solution development and validation, with some later stages showing higher confirmed issue-resolution proportions. Trajectory analysis identifies formulation and implementation failures, while task-equalized research-chain completion rates range from 24% to 45% across models on evaluable tasks. These findings motivate sustained information-gap resolution, faithful problem formulation, and evidence-linked self-improvement for reliable long-horizon agents.

Abstract of OR-License: Evaluating LLM Agents as Real-World Operations Research Practitioners (arXiv:2610.24087, CC0)
Made by ALLE AI Released Oct 2026 Tasks from real jobs All tasks public Last model added Sep 2026 Scores refreshed 2 Nov 2026
Level
Expert Specialists with years of training, often over hours of work
Scored
Graded by people Human experts grade or compare the outputs.
Answers
Agent task Many steps in a terminal, browser, desktop, app, game or robot.
11
Entries on its own leaderboard, 10 models we track

Grok 4.6 leads with 88.4% among models released by Sep 2026

Best score per entry from the benchmark's own leaderboard, read on 2 Nov 2026. Where a model ran with several settings or harnesses, we keep its best. Names in gray aren't in our model list: fine-tunes, agent systems, products and older models. They don't count toward leaders.

The newest model with a score here came out in Sep 2026, and nobody has tested a newer one. Among the 20 highest-ranked models on the capability index, these have no score yet:

  1. 1 Grok 4.6 high 88.4%
  2. 2 GPT-5.6 Sol xhigh 84.5%
  3. 3 Gemini 3.8 Flash high 82.6%
  4. 4 Qwen 3.8 Max xhigh 78.7%
  5. 5 Claude Opus 5 xhigh 77.4%
  6. 6 Muse Spark 1.3 high 74.2%
  7. 7 DeepSeek V4.1 Flash high 69.7%
  8. 8 Tencent Hy4 preview high 66.5%
  9. 9 Doubao-Seed-2.1-Pro high 65.2%
  10. 10 GLM-5.3-Flash max 62.6%
  11. 11 Tencent Hy3 high 45.2%
#EntryScoreSettingDateSource
1 Grok 4.6 88.4% high Aug 2026 Official leaderboard
2 GPT-5.6 Sol 84.5% xhigh Jul 2026 Official leaderboard
3 Gemini 3.8 Flash 82.6% high Sep 2026 Official leaderboard
4 Qwen 3.8 Max 78.7% xhigh Aug 2026 Official leaderboard
5 Claude Opus 5 77.4% xhigh Jul 2026 Official leaderboard
6 Muse Spark 1.3 74.2% high Sep 2026 Official leaderboard
7 DeepSeek V4.1 Flash 69.7% high Sep 2026 Official leaderboard
8 Tencent Hy4 preview 66.5% high Aug 2026 Official leaderboard
9 Doubao-Seed-2.1-Pro 65.2% high Official leaderboard
10 GLM-5.3-Flash 62.6% max Aug 2026 Official leaderboard
11 Tencent Hy3 45.2% high Jul 2026 Official leaderboard

Number one over time

Lab claims count from the day the card came out, independent runs from the model’s release.

xAI
Own leaderboard Passed 85% in Oct 2026, by Grok 4.6
Grok 4.6 88.4%

The best score passed 85% in Aug 2026

All the scores we have, one dot each, so a model tested at several settings appears more than once.

On the benchmark’s own leaderboard Best so far
0% 20% 40% 60% 80% 100% Near the ceiling (85%) Grok 4.6 GPT-5.6 Sol Tencent Hy3 0% 20% 40% 60% 80% 100% Near the ceiling (85%) Grok 4.6 Tencent Hy3

No lab model card reports it yet

We read the headline results of 479 launch posts from 51 labs, and none of them list this benchmark.

More in Economic work

All 91 in Economic work
  • JobBench In use 130 tasks from 35 occupations workers most want delegated. Agent works through messy reference files, graded by fact-anchored binary rubrics. GLM-5.3 61.4%
  • GDPval In use Real work tasks from 44 occupations with reference files. Experts blindly compare model deliverables with expert work. Metric is win-or-tie rate. Seed2.1 Pro 87.9%
  • GDPval-AA v2.1 In use Agents with shell and web access complete 220 GDPval gold tasks from 44 occupations. Blind pairwise comparisons give Elo ratings. Claude Opus 5.5 1,846
  • AA-Briefcase v1.1 In use Version 1.1 of Artificial Analysis's multi-hour office work evaluation, with Elo ratings fitted by Crowd-BT. Claude Opus 5.5 1,822
  • CoWorkBench In use Agent completes long-horizon productivity tasks across computer science, finance, law and medicine. Qwen in-house scoring (method unpublished). Qwen3.8-Omni-Flash 75.3%
  • OfficeQA In use Grounded reasoning over a large corpus of U.S. Treasury Bulletin documents: locate tables and compute precise numeric answers. Claude Opus 4.7 86.3%