IHEval

Nemotron 3.5 Bolt E4B 82.4% leads since Oct 2026 In use

Models face conflicts between system, user and tool instructions. Reports a composite score for following the higher-priority instruction.

Instruction hierarchy benchmark. Meta reports a composite across conflict types.

In the authors’ words

The instruction hierarchy, which establishes a priority order from system messages to user messages, conversation history, and tool outputs, is essential for ensuring consistent and safe behavior in language models (LMs). Despite its importance, this topic receives limited attention, and there is a lack of comprehensive benchmarks for evaluating models' ability to follow the instruction hierarchy. We bridge this gap by introducing IHEval, a novel benchmark comprising 3,538 examples across nine tasks, covering cases where instructions in different priorities either align or conflict. Our evaluation of popular LMs highlights their struggle to recognize instruction priorities. All evaluated models experience a sharp performance decline when facing conflicting instructions, compared to their original instruction-following performance. Moreover, the most competitive open-source model only achieves 48% accuracy in resolving such conflicts. Our results underscore the need for targeted optimization in the future development of LMs.

Abstract of IHEval: Evaluating Language Models on Following the Instruction Hierarchy (arXiv:2502.08745, CC0)

Made by Zhang et al. (Notre Dame + Amazon) Released Feb 2025 Jailbreaks and prompt injection All tasks public Last in a launch post Oct 2026 Last model added Oct 2026

Level: Everyday Most adults, without training
Scored: Checked automatically A program checks the result against an answer key, unit tests or the expected final state.
Answers: Open-ended Free-form text, documents, designs or media judged as a whole.

ytyz1307zzh/IHEval 19 stars Paper

Models scored in lab cards

Launch post lists it, from 1 lab

System cards report it, from 2 labs

Nemotron 3.5 Bolt E4B leads with 82.4%

Best score per model as labs report them in their own cards. Labs pick their own settings, so these compare less cleanly than independent runs.

Bars Table

1 Nemotron 3.5 Bolt E4B 82.4%
2 Muse Spark 80.3%

#	Model	Score	Setting	Date	Source
1	Nemotron 3.5 Bolt E4B	82.4%		Oct 2026	Model card
2	Muse Spark	80.3%		Jun 2026	Model card

2 models have held number one since its release in Feb 2025

Lab claims count from the day the card came out, independent runs from the model’s release.

Meta NVIDIA

Lab claims 2 leaders

Nemotron 3.5 Bolt E4B 82.4%

Scores over time

All the scores we have, one dot each, so a model tested at several settings appears more than once.

Reported by the lab in its model card Best so far

One lab put it in a launch post in the last six months

Launch posts from every lab that has listed it, one tick per post. Tall colored ticks include it, and short gray ones leave it out.

NVIDIA

Date	Launch post	Score	As the post writes it
23 Oct 2026	Nemotron 3.5 Bolt E4B	82.4	IHEval

2 system cards report it, from 2 labs

Besides launch posts, we read the system cards, model cards and technical reports labs publish with a release. They report far more, including safety and frontier-risk tests.

Date	Document	Score	As the document writes it	Section
23 Oct 2026	Nemotron 3.5 Bolt E4B technical report	82.4	IHEval, released Q4_K_M imatrix checkpoint (Table 5). BF16 84.79 (Table 3)	Capabilities
12 Jun 2026	Muse Spark system card	80.3	IHEval composite score (Table 2, updated after the evaluation bug fix noted in the change log)	Alignment

Nemotron 3.5 Bolt E4B leads with 82.4%

2 models have held number one since its release in Feb 2025

Scores over time

One lab put it in a launch post in the last six months

2 system cards report it, from 2 labs

More in Safety and alignment