Benchmark Almanac

Llama-3.1-Nemotron-Ultra-253B-v1

Llama-3.1-Nemotron-Ultra-253B-v1 places in the top three on BigO(Bench).

NVIDIA Released 7 Apr 2025 Family Llama Nemotron The lab's model card
13
Benchmarks with a score
0
Ranked first, out of five or more models
1
In the top three
NVIDIA Apr 2025

Llama-3.1-Nemotron-Ultra-253B-v1

Open weights Flagship

No Epoch Capabilities Index score yet
  • LiveCodeBench v5 66.3% #15 /44
  • BigO(Bench) 41.1% #3 /18
  • AIME 2025 72.5% #106 /143

Where it ranks

Its best score on each benchmark, and where that ranks among every model with a score there. We rank each model once, at its best setting.

Knowledge and language

BenchmarkScoreRankSource
IFEval 89.5% #32 of 126 Model card
Arena-Hard-Auto 87% #15 of 39 Model card
JudgeBench 79.1% #1 of 2 Model card
LMArena (Elo) 1,348 #178 of 333 Official leaderboard
EQ-Bench 3 1,093 #54 of 78 Official leaderboard

Math and science

BenchmarkScoreRankSource
MATH-500 97% #19 of 67 Model card
AIME 2024 80.8% #39 of 87 Model card
GPQA Diamond 76% #136 of 284 Model card
AIME 2025 72.5% #106 of 143 Model card

Coding

BenchmarkScoreRankSource
BigO(Bench) 41.1% #3 of 18 Official leaderboard
LiveCodeBench v5 66.3% #15 of 44 Model card

Agents and tools

BenchmarkScoreRankSource
Berkeley Function Calling Leaderboard (BFCL) 74.1% #11 of 47 Model card
BFCL V4 10% #64 of 65 Official leaderboard

Its launch post

The benchmarks the lab chose to headline, with the scores it gave.

Its system card

Every result in the longer documents the lab published with it. 2 of them aren't in the launch post.