Code Interpreter Benchmark Math

Qwen1.5-72B 73.1%

Qwen-14B 58.4% leads

Qwen1.5-72B 73.1% leads since February 2024

Dropped

Labs’ own numbers All independent runs Include claimed scores

Models use a Python interpreter to solve math problems. The score is execution accuracy on the Math task set.

One track of Code Interpreter Benchmark. The Code Interpreter Benchmark record keeps lab scores that combine tracks.

Made by: Qwen
Released: September 2023
Kind: Function calling
Level: School
Scored: Checked automatically
Answers: Agent task
Tasks: All public

Strongest model tested: Qwen-14B #213 on the capability index
Last in a launch post: February 2024, dropped before it saturated
Last model added: February 2024
Scores refreshed: November 12th, 2026

QwenLM/Qwen-Agent 17,261 stars

Models scored in lab cards

Launch posts list it, from 1 lab

Qwen1.5-72B leads with 73.1%

Best score per model as labs report them in their own cards. Labs pick their own settings, so these compare less cleanly than independent runs.

Qwen-14B leads with 58.4% across all independent runs

Best score per model across the official leaderboard, Epoch AI’s own runs and any board Epoch AI copied. Each bar and row names its source.

Qwen1.5-72B leads with 73.1%, claims included

Best score per model from every source, including labs’ own numbers and claims Epoch AI collected from reports and papers. Labs pick their own settings, so a claim can beat an independent run of the same model.

Bars Table

1 Qwen1.5-72B 73.1%
2 Qwen-14B 58.4%

Source: Labs’ model cards

1 Qwen-14B version 20231206 58.4%
2 ChatGLM3-6B version 20231206 54.2%

Source: Official Code Interpreter Benchmark Math leaderboard

Sources: Labs’ model cards and official Code Interpreter Benchmark Math leaderboard

#	Model	Score	Setting	Date	Source
1	Qwen1.5-72B	73.1%		Feb 2024	Lab model card
2	Qwen-14B	58.4%		Sep 2023	Lab model card

#	Model	Score	Setting	Date	Source
1	Qwen-14B	58.4%	version 20231206	Sep 2023	Official leaderboard
2	ChatGLM3-6B	54.2%	version 20231206	Oct 2023	Official leaderboard

#	Model	Score	Setting	Date	Source
1	Qwen1.5-72B	73.1%		Feb 2024	Lab model card
2	Qwen-14B	58.4%	version 20231206	Sep 2023	Official leaderboard
3	ChatGLM3-6B	54.2%	version 20231206	Oct 2023	Official leaderboard

The benchmark's own leaderboard has 10 entries, 2 of them models we track. Switch the scores above to rank them together.

Scores over time

All the scores we have, one dot each, so a model tested at several settings appears more than once.

On the benchmark’s own leaderboard Reported by the lab in its model card Best so far

Sources: Official Code Interpreter Benchmark Math leaderboard and labs’ model cards

No launch post has listed it since February 2024

Launch posts from every lab that has listed it, one tick per post. Tall colored ticks include it, and short gray ones leave it out.

Qwen

Date	Launch post	Score	As the post writes it
Feb 5, 2024	Qwen1.5-72B	73.1	Code Interpreter Benchmark (Accuracy of Code Execution Results, Math)
Sep 25, 2023	Qwen-14B	58.4	Code Interpreter benchmark, Accuracy of Code Execution Results (Math)

One lab claim next to the leaderboard

Each score a lab reported for its own model, beside the same model's best score on the leaderboard. Gaps mostly come from different harnesses, effort levels and task subsets. 1 of them match, which usually means one side copied the other. Every benchmark.

Model	Lab says	Board says	Gap
Qwen-14B	58.4% Code Interpreter benchmark, Accuracy of Code Execution Results (Math)	58.4% Leaderboard, version 20231206	Same