SWE-Perf

Agents speed up code in 140 tasks drawn from performance pull requests in real GitHub repos. Scored on correctness and runtime gain.

In the authors’ words

Code performance optimization is paramount in real-world software engineering and critical for production-level systems. While Large Language Models (LLMs) have demonstrated impressive capabilities in code generation and bug fixing, their proficiency in enhancing code performance at the repository level remains largely unexplored. To address this gap, we introduce SWE-Perf, the first benchmark specifically designed to systematically evaluate LLMs on code performance optimization tasks within authentic repository contexts. SWE-Perf comprises 140 carefully curated instances, each derived from performance-improving pull requests from popular GitHub repositories. Each benchmark instance includes the relevant codebase, target functions, performance-related tests, expert-authored patches, and executable environments. Through a comprehensive evaluation of representative methods that span file-level and repo-level approaches (e.g., Agentless and OpenHands), we reveal a substantial capability gap between existing LLMs and expert-level optimization performance, highlighting critical research opportunities in this emerging field.

Abstract of SWE-Perf: Can Language Models Optimize Code Performance on Real-World Repositories? (arXiv:2507.12415, CC0)

Made by TikTok and Xi'an Jiaotong University Released Jul 2025 Speed and optimization All tasks public Last in a launch post Dec 2025 , dropped before it saturated Last model added Dec 2025

Level: Expert Specialists with years of training, often over hours of work
Scored: Checked automatically A program checks the result against an answer key, unit tests or the expected final state.
Answers: Code A program or patch run against tests.

SWE-Perf/SWE-Perf 55 stars Paper Dataset swe-perf.github.io

Model scored in lab cards

Launch post lists it, from 1 lab

MiniMax-M2.1 leads with 3.10%

Best score per model as labs report them in their own cards. Labs pick their own settings, so these compare less cleanly than independent runs.

Bars Table

3.10 MiniMax-M2.1

1 MiniMax-M2.1 3.10%

#	Model	Score	Setting	Date	Source
1	MiniMax-M2.1	3.10%		Dec 2025	Model card

Scores over time

All the scores we have, one dot each, so a model tested at several settings appears more than once.

Reported by the lab in its model card

Labs headline it less often now, last in Dec 2025

Launch posts from every lab that has listed it, one tick per post. Tall colored ticks include it, and short gray ones leave it out.

MiniMax

Date	Launch post	Score	As the post writes it
23 Dec 2025	MiniMax-M2.1	3.1	SWE-Perf

MiniMax-M2.1 leads with 3.10%

Scores over time

Labs headline it less often now, last in Dec 2025

More in Coding