SWE-bench Verified

Coding · fixed question set

500 real GitHub issues from Python repositories, human-checked to be solvable. The model must produce a patch that passes the repository's own tests.

How far to trust it

🚨 A SWE-bench number is never a fact about a model alone. It is a fact about a model inside an agent scaffold, and the scaffold changes the score. Every figure here records which scaffold ran. One audit of the top entries found about 19.78% of 'solved' cases were semantically wrong: the patch passed the tests without fixing the issue. Read this benchmark as a floor on capability, never as a percentage of real bugs fixed.

Measured

Models scored

Spread between models

20.4

standard deviation, points

Measurement noise

2.0

published stderr

Weight

0.99

share of spread that is signal

The weight is (spread² − noise²) / spread². It is computed, never chosen. Most of this benchmark's spread is real, so it still separates models.

Every model scored

the median across configurations, so one heroic run cannot lead

Rank	Model	Lab	Score
1	Claude Opus 4.7	Anthropic	83.5
2	GPT-5.5	OpenAI	80.6
3	Doubao-Seed-Code	ByteDance	78.8
4	GLM-5.2	Zhipu	78.7
5	DeepSeek v4	DeepSeek	77.6
6	Qwen3.7-Max	Alibaba	77.3
7	Claude Opus 4.6	Anthropic	77.2
8	GPT-5.4	OpenAI	76.9
9	Claude 4.5 Opus	Anthropic	76.8
10	Claude Opus 4.5	Anthropic	76.7
11	Kimi K2.6	Kimi	76.7
12	Qwen3.6 Max Preview	Alibaba	76.7
13	MiniMax M2.5	MiniMax	75.8
14	Gemini 3.1 Pro Preview	Google	75.6
15	Gemini 3 Flash Preview	Google	75.6
16	Claude 4.6 Opus	Anthropic	75.6
17	Gemini 3.5 Flash	Google	75.6
18	Claude Sonnet 4.6	Anthropic	75.2
19	GPT-5.3 Codex	OpenAI	74.8
20	GLM-5.1	Zhipu	74.2
21	Claude Opus 4.1	Anthropic	73.3
22	Gemini 3 Pro Preview	Google	72.9
23	GPT 5.2 Codex	OpenAI	72.8
24	GLM 5	Zhipu	72.4
25	GPT 5.2	OpenAI	72.3
26	Kimi K2.5	Kimi	72.3
27	Claude 4.5 Sonnet	Anthropic	72.0
28	GPT 5	OpenAI	71.6
29	Claude Sonnet 4.5	Anthropic	71.3
30	Claude 4 Sonnet	Anthropic	71.0
31	Claude Opus 4	Anthropic	70.7
32	Claude 4 Opus	Anthropic	70.4
33	Claude 4.5 Haiku	Anthropic	66.6
34	GPT 5.1	OpenAI	66.5
35	GPT 5.1 Codex	OpenAI	66.0
36	Claude Sonnet 4	Anthropic	65.9
37	DeepSeek-V3.2-Exp	DeepSeek	65.0
38	o1-2024-12-17	OpenAI	64.6
39	Multiple	Anthropic	64.1
40	Claude 3.7 Sonnet w/ Review Heavy	Anthropic	62.4
41	swe-search	Anthropic	62.2
42	MiniMax M2	MiniMax	61.0
43	o3	OpenAI	60.4
44	Claude 3.7 Sonnet	Anthropic	60.3
45	GPT 5 mini	OpenAI	59.8
46	Kimi K2 Instruct	Kimi	59.4
47	TTS	OpenAI	58.8
48	Qwen3.6 Plus	Alibaba	57.9
49	Devstral Small	Mistral	56.4
50	Qwen3-Coder-30B-A3B-Instruct	Alibaba	56.3
51	Gemini 2.5 Pro	Google	55.6
52	GLM 4.6	Zhipu	55.4
53	Qwen3-Coder 480B/A35B Instruct	Alibaba	55.4
54	o4-mini	OpenAI	54.8
55	GLM 4.5	Zhipu	54.2
56	Devstral	Mistral	53.8
57	Frogboss 32B 2510	Microsoft	53.6
58	Claude 3.5 Sonnet	Anthropic	50.7
59	TTS	Alibaba	47.0
60	DevStral Small 2505	Mistral	46.8
61	Frogmini 14B 2510	Microsoft	45.0
62	GPT 4.1	OpenAI	44.1
63	Amazon.nova Premier v1:0	Amazon	42.4
64	O3 Mini	OpenAI	42.4
65	DeepSWE-Preview	Agentica	42.2
66	Llama3-SWE-RL-70B	Meta	41.2
67	Claude 3.5 Haiku	Anthropic	40.6
68	SWE-agent-LM-32B	Alibaba	40.2
69	DevStral Small 2507	Mistral	38.0
70	GPT 5 nano	OpenAI	34.8
71	Qwen2.5	Alibaba	31.5
72	GPT 4o	OpenAI	30.0
73	Gemini 2.0 Flash Thinking	Google	28.9
74	Gemini 2.5 Flash	Google	28.7
75	gpt-oss-120b	OpenAI	26.0
76	GPT 4.1 mini	OpenAI	23.9
77	Qwen2.5 Coder 32B Instruct	Alibaba	23.5
78	MCTS Refine 7B	—	23.2
79	Llama 4 Maverick Instruct	Meta	21.0
80	GPT-4	OpenAI	12.6
81	Claude 3 Opus	Anthropic	11.4
82	Llama 4 Scout Instruct	Meta	9.1
83	Claude 2	Anthropic	4.4
84	SWE-Llama 7B	Meta	1.4

The benchmark's own page