OTIS Mock AIME

Mathematics · questions rotate

Mock AIME problems written for the OTIS programme, answered as an integer from 0 to 999.

How far to trust it

Written after the models were trained, and not a public exam set, so contamination is low. The integer answer removes grader judgement. It carries the widest spread between models of any benchmark here, which makes it the strongest single discriminator.

Measured

Models scored

213

Spread between models

35.7

standard deviation, points

Measurement noise

4.8

published stderr

Weight

0.98

share of spread that is signal

The weight is (spread² − noise²) / spread². It is computed, never chosen. Most of this benchmark's spread is real, so it still separates models.

Every model scored

the median across configurations, so one heroic run cannot lead

Rank	Model	Lab	Score
1	Claude Fable 5.1	Anthropic	100.0
2	Claude Opus 5.5	Anthropic	100.0
3	Claude Sonnet 5.5	Anthropic	100.0
4	Gemini 4 Argon	Google	100.0
5	GPT-5.5 Pro pre-release	OpenAI	100.0
6	GPT-6.1 Sol	OpenAI	100.0
7	GPT-6 Astra	OpenAI	100.0
8	GPT-6 Sol	OpenAI	100.0
9	Qwen3.8 Max (0902)	Alibaba	100.0
10	Claude Fable 5	Anthropic	99.7
11	Muse Wonder	Meta	99.4
12	Qwen3.8 Max	Alibaba	99.4
13	Muse Spark 1.3	Meta	99.2
14	Gemini 3.8 Flash	Google	98.9
15	GPT-6 Luna	OpenAI	98.9
16	DeepSeek V4 Pro 0813	DeepSeek	98.6
17	Grok 4.6	xAI	98.5
18	DeepSeek V4.1 Flash	DeepSeek	98.3
19	Mistral Large 4	Mistral	98.3
20	Grok 4.7	xAI	98.1
21	Muse Spark Open	Meta	98.1
22	Claude Haiku 5.5	Anthropic	98.1
23	Claude Opus 4.8	Anthropic	98.1
24	Claude Opus 5	Anthropic	97.8
25	GPT-5.6 Sol	OpenAI	97.8
26	Grok 4.5	xAI	97.8
27	Gemini 3.7 Flash	Google	97.2
28	DeepSeek v4	DeepSeek	96.1
29	Kimi K2.6	Kimi	96.1
30	Gemini 3.1 Pro Preview	Google	95.6
31	Kimi K2.7 Code	Kimi	95.6
32	Qwen3.7-Max	Alibaba	95.6
33	GPT-5.4	OpenAI	95.4
34	GPT 5.2	OpenAI	95.0
35	DeepSeek V4 Flash 0731	DeepSeek	94.4
36	GPT-5.6 Terra	OpenAI	94.3
37	Gemini 3 Flash Preview	Google	94.2
38	GLM-5.3-Flash	Zhipu	93.9
39	GLM-5.1	Zhipu	93.3
40	Grok 4.3 Beta	xAI	93.3
41	Kimi K3	Kimi	93.3
42	Qwen3.6 Plus	Alibaba	93.3
43	Qwen3.7 Plus	Alibaba	93.3
44	Claude Opus 4.6	Anthropic	93.1
45	Claude Opus 4.7	Anthropic	92.2
46	GPT-5.5	OpenAI	92.2
47	Grok 4.20	xAI	92.2
48	Kimi K2.5	Kimi	92.2
49	Gemini 3 Pro Preview	Google	91.4
50	GLM-5.3	Zhipu	91.1
51	Qwen3.6 Max Preview	Alibaba	91.1
52	Qwen3.6 27B	Alibaba	91.1
53	Inkling Small	Thinking Machines	90.0
54	Muse Spark	Meta	88.9
55	Gemini 3.5 Flash	Google	88.9
56	gpt-oss-120b	OpenAI	88.9
57	Inkling	Thinking Machines	88.9
58	Qwen3.5 397B-A17B	Alibaba	88.9
59	GPT-5.4 mini	OpenAI	88.1
60	Claude Sonnet 5	Anthropic	87.4
61	GPT 5	OpenAI	87.2
62	Nemotron 3 Ultra	NVIDIA	86.7
63	Qwen3-235B-A22B	Alibaba	86.7
64	Qwen3.5 Plus	Alibaba	86.7
65	Qwen3.6 35B-A3B	Alibaba	86.7
66	Qwen3.7 Flash	Alibaba	86.7
67	GPT 5.1	OpenAI	85.6
68	Claude Opus 4.8	Anthropic	84.4
69	Qwen3.5 Flash	Alibaba	84.4
70	Qwen3.6 Flash	Alibaba	84.4
71	Gemini 2.5 Pro	Google	84.2
72	Grok 4	xAI	84.0
73	Claude Opus 4.5	Anthropic	83.9
74	GLM-4.7	Zhipu	83.3
75	Kimi K2 Instruct	Kimi	83.1
76	GPT-5.6 Luna	OpenAI	82.5
77	Gemini 3.6 Flash	Google	82.2
78	Gemma 4 26B-A4B IT	Google	82.2
79	Qwen3.5 397B-A17B	Alibaba	82.2
80	GLM-5.2	Zhipu	81.0
81	GLM 5	Zhipu	80.0
82	Qwen3.7 Plus	Alibaba	80.0
83	Claude Sonnet 4.6	Anthropic	78.9
84	GPT-5.4 nano	OpenAI	78.3
85	Claude Sonnet 4.5	Anthropic	77.8
86	Qwen3.7 Flash	Alibaba	77.8
87	GPT 5 mini	OpenAI	76.8
88	o3	OpenAI	76.1
89	GPT 5 nano	OpenAI	74.2
90	Gemma 4 31B IT	Google	73.3
91	Qwen3 Max	Alibaba	73.3
92	Gemini 2.5 Flash	Google	71.9
93	MiniMax M3	MiniMax	71.1
94	o4-mini	OpenAI	70.9
95	Grok-3 mini	xAI	70.0
96	GPT-5.6 Sol	OpenAI	68.9
97	Qwen3.6 35B-A3B	Alibaba	68.9
98	DeepSeek-V3.2-Exp	DeepSeek	68.4
99	GPT-5.5 Instant	OpenAI	68.1
100	seed-oss-36b-instruct	—	67.5
101	Qwen3.6 27B	Alibaba	66.7
102	DeepSeek-R1	DeepSeek	66.4
103	Qwen3-30B-A3B	Alibaba	66.2
104	Claude Opus 4.1	Anthropic	64.4
105	GPT-5.2	OpenAI	62.2
106	Qwen3.5-35B-A3B	Alibaba	62.2
107	O3 Mini	OpenAI	61.8
108	Qwen3.5 9B	Alibaba	61.7
109	Claude Sonnet 4	Anthropic	61.1
110	Claude Opus 4	Anthropic	60.0
111	Gemini 3.5 Flash-Lite	Google	60.0
112	QwQ 32B	Alibaba	59.2
113	glm-4.7-flash	—	58.3
114	GPT-5.4	OpenAI	57.8
115	GPT-5.5	OpenAI	57.8
116	o1-2024-12-17	OpenAI	57.8
117	Qwen3 8B	Alibaba	56.1
118	Qwen3.5-4B	Alibaba	55.8
119	DeepSeek-R1-Distill-Qwen-32B	DeepSeek	55.6
120	Grok 3 Beta	xAI	55.6
121	DeepSeek R1	DeepSeek	53.3
122	GPT-5.6 Terra	OpenAI	53.3
123	qwen3-4b-instruct-2507	—	52.2
124	DeepSeek-R1-Distill-Llama-70B	DeepSeek	51.4
125	Claude Haiku 4.5	Anthropic	51.2
126	gpt-oss-20b	OpenAI	50.8
127	DeepSeek-R1-Distill-Qwen-14B	DeepSeek	50.6
128	Claude 3.7 Sonnet	Anthropic	50.0
129	Claude Opus 4.5	Anthropic	48.1
130	DeepSeek v4	DeepSeek	46.7
131	GPT-5.4 nano	OpenAI	46.7
132	Qwen3-14B	Alibaba	46.1
133	o1-mini	OpenAI	45.8
134	Qwen3 32B	Alibaba	45.0
135	GPT 4.1 mini	OpenAI	44.7
136	Gemini 2.0 Flash Thinking	Google	44.4
137	Gemini 3.1 Flash-Lite	Google	44.4
138	Qwen3-30B-A3B	Alibaba	44.2
139	DeepSeek-R1-0528	—	43.9
140	Qwen3.5 9B	Alibaba	42.8
141	GPT-5.6 Luna	OpenAI	40.0
142	GPT 4.1	OpenAI	38.3
143	DeepSeek-V3	DeepSeek	37.8
144	GPT-4.5	OpenAI	37.8
145	GPT-5.1	OpenAI	37.8
146	Claude Sonnet 4.5	Anthropic	35.6
147	Mistral Medium 3	Mistral	32.2
148	Mistral Small 3.2	Mistral	30.3
149	Magistral Small 1.0	Mistral	30.0
150	GLM-5.2	Zhipu	28.9
151	GPT-4.1 nano	OpenAI	28.9
152	Magistral Small 1.2	Mistral	28.1
153	GPT-5.4 mini	OpenAI	26.7
154	MiniMax M3	MiniMax	26.7
155	glm-4.7-flash_none	—	25.0
156	Gemma 3 27B	Google	22.5
157	Qwen3 8B	Alibaba	22.2
158	DeepSeek-R1-Distill-Qwen-1.5B	DeepSeek	21.4
159	Llama 4 Maverick Instruct	Meta	20.6
160	Qwen Plus	Alibaba	17.8
161	Gemma 3 12B	Google	16.7
162	Qwen2.5-Max	Alibaba	16.1
163	DeepSeek-V3	DeepSeek	15.8
164	Gemini 1.5 Pro	Google	14.9
165	Phi-4	Microsoft	13.8
166	Grok-2	xAI	11.5
167	Gemini 1.5 Flash	Google	10.1
168	Llama 3.1-405B	Meta	9.7
169	Claude 3.5 Sonnet	Anthropic	8.5
170	Mistral Large 2	Mistral	8.1
171	Qwen2.5-72B	Alibaba	8.1
172	Qwen3-1.7B	Alibaba	8.1
173	Llama 4 Scout Instruct	Meta	7.8
174	Gemma 3 4B	Google	7.5
175	Qwen2.5-32B	Alibaba	7.4
176	GPT-4o mini	OpenAI	6.9
177	GPT-4 Turbo	OpenAI	6.7
178	Claude 3.5 Sonnet	Anthropic	6.5
179	GPT 4o	OpenAI	6.3
180	Qwen Turbo	Alibaba	6.1
181	Mistral Small 3	Mistral	6.0
182	Llama 3.3 70B	Meta	5.1
183	Mistral Small 3.1	Mistral	4.9
184	Claude 3 Opus	Anthropic	4.7
185	Gemini 1.5 Flash 8B	Google	4.6
186	Tülu 3 70B	Ai2	4.4
187	Claude 3.5 Haiku	Anthropic	4.3
188	Llama 3-70B	Meta	4.3
189	Llama 3.1-70B	Meta	3.6
190	granite-4.0-micro	—	2.8
191	Llama 3.2 90B	Meta	2.6
192	Claude 2	Anthropic	2.5
193	Claude 3 Sonnet	Anthropic	2.5
194	Hermes 2 Theta Llama-3 70B	Nous Research	2.5
195	Qwen2.5-7B	Alibaba	2.5
196	GPT-3.5 Turbo	OpenAI	2.2
197	Claude 2.1	Anthropic	1.9
198	Llama 3-8B	Meta	1.9
199	Mistral Large	Mistral	1.9
200	Claude 3 Haiku	Anthropic	1.8
201	Llama 3.1-8B	Meta	1.7
202	Gemma 2 27B	Google	1.4
203	Gemini 1.0 Pro	Google	1.1
204	Gemma 3 1B	Google	1.1
205	GPT-4	OpenAI	1.1
206	DeepSeek LLM 67B	DeepSeek	0.8
207	granite-4.0-1b	—	0.8
208	Gemma 2 9B	Google	0.6
209	GPT-4	OpenAI	0.6
210	Llama 3.2 1B	Meta	0.6
211	granite-4.0-350m	—	0.3
212	Mistral 7B	Mistral	0.3
213	Llama 2-70B	Meta	0.0

The benchmark's own page