OTIS Mock AIME

Mathematics · questions rotate

Mock AIME problems written for the OTIS programme, answered as an integer from 0 to 999.

How far to trust it

Written after the models were trained, and not a public exam set, so contamination is low. The integer answer removes grader judgement. It carries the widest spread between models of any benchmark here, which makes it the strongest single discriminator.

Measured

Models scored

200

Spread between models

35.1

standard deviation, points

Measurement noise

4.9

published stderr

Weight

0.98

share of spread that is signal

The weight is (spread² − noise²) / spread². It is computed, never chosen. Most of this benchmark's spread is real, so it still separates models.

Every model scored

the median across configurations, so one heroic run cannot lead

Rank	Model	Lab	Score
1	Claude Fable 5.1	Anthropic	100.0
2	GPT-5.5 Pro pre-release	OpenAI	100.0
3	GPT-6 Astra	OpenAI	100.0
4	Qwen3.8 Max (0902)	Alibaba	100.0
5	Claude Fable 5	Anthropic	99.7
6	Qwen3.8 Max	Alibaba	99.4
7	Gemini 3.8 Flash	Google	98.9
8	DeepSeek V4 Pro 0813	DeepSeek	98.6
9	Grok 4.6	xAI	98.5
10	Claude Opus 4.8	Anthropic	98.1
11	Claude Opus 5	Anthropic	97.8
12	GPT-5.6 Sol	OpenAI	97.8
13	Grok 4.5	xAI	97.8
14	Gemini 3.7 Flash	Google	97.2
15	DeepSeek v4	DeepSeek	96.1
16	Kimi K2.6	Kimi	96.1
17	Gemini 3.1 Pro Preview	Google	95.6
18	Kimi K2.7 Code	Kimi	95.6
19	Qwen3.7-Max	Alibaba	95.6
20	GPT-5.4	OpenAI	95.4
21	GPT 5.2	OpenAI	95.0
22	DeepSeek V4 Flash 0731	DeepSeek	94.4
23	GPT-5.6 Terra	OpenAI	94.3
24	Gemini 3 Flash Preview	Google	94.2
25	GLM-5.3-Flash	Zhipu	93.9
26	GLM-5.1	Zhipu	93.3
27	Grok 4.3 Beta	xAI	93.3
28	Kimi K3	Kimi	93.3
29	Qwen3.6 Plus	Alibaba	93.3
30	Qwen3.7 Plus	Alibaba	93.3
31	Claude Opus 4.6	Anthropic	93.1
32	Claude Opus 4.7	Anthropic	92.2
33	GPT-5.5	OpenAI	92.2
34	Grok 4.20	xAI	92.2
35	Kimi K2.5	Kimi	92.2
36	Gemini 3 Pro Preview	Google	91.4
37	GLM-5.3	Zhipu	91.1
38	Qwen3.6 Max Preview	Alibaba	91.1
39	Qwen3.6 27B	Alibaba	91.1
40	Inkling Small	Thinking Machines	90.0
41	Muse Spark	Meta	88.9
42	Gemini 3.5 Flash	Google	88.9
43	gpt-oss-120b	OpenAI	88.9
44	Inkling	Thinking Machines	88.9
45	Qwen3.5 397B-A17B	Alibaba	88.9
46	GPT-5.4 mini	OpenAI	88.1
47	Claude Sonnet 5	Anthropic	87.4
48	GPT 5	OpenAI	87.2
49	Nemotron 3 Ultra	NVIDIA	86.7
50	Qwen3-235B-A22B	Alibaba	86.7
51	Qwen3.5 Plus	Alibaba	86.7
52	Qwen3.6 35B-A3B	Alibaba	86.7
53	Qwen3.7 Flash	Alibaba	86.7
54	GPT 5.1	OpenAI	85.6
55	Claude Opus 4.8	Anthropic	84.4
56	Qwen3.5 Flash	Alibaba	84.4
57	Qwen3.6 Flash	Alibaba	84.4
58	Gemini 2.5 Pro	Google	84.2
59	Grok 4	xAI	84.0
60	Claude Opus 4.5	Anthropic	83.9
61	GLM-4.7	Zhipu	83.3
62	Kimi K2 Instruct	Kimi	83.1
63	GPT-5.6 Luna	OpenAI	82.5
64	Gemini 3.6 Flash	Google	82.2
65	Gemma 4 26B-A4B IT	Google	82.2
66	Qwen3.5 397B-A17B	Alibaba	82.2
67	GLM-5.2	Zhipu	81.0
68	GLM 5	Zhipu	80.0
69	Qwen3.7 Plus	Alibaba	80.0
70	Claude Sonnet 4.6	Anthropic	78.9
71	GPT-5.4 nano	OpenAI	78.3
72	Claude Sonnet 4.5	Anthropic	77.8
73	Qwen3.7 Flash	Alibaba	77.8
74	GPT 5 mini	OpenAI	76.8
75	o3	OpenAI	76.1
76	GPT 5 nano	OpenAI	74.2
77	Gemma 4 31B IT	Google	73.3
78	Qwen3 Max	Alibaba	73.3
79	Gemini 2.5 Flash	Google	71.9
80	MiniMax M3	MiniMax	71.1
81	o4-mini	OpenAI	70.9
82	Grok-3 mini	xAI	70.0
83	GPT-5.6 Sol	OpenAI	68.9
84	Qwen3.6 35B-A3B	Alibaba	68.9
85	DeepSeek-V3.2-Exp	DeepSeek	68.4
86	GPT-5.5 Instant	OpenAI	68.1
87	seed-oss-36b-instruct	—	67.5
88	Qwen3.6 27B	Alibaba	66.7
89	DeepSeek-R1	DeepSeek	66.4
90	Qwen3-30B-A3B	Alibaba	66.2
91	Claude Opus 4.1	Anthropic	64.4
92	GPT-5.2	OpenAI	62.2
93	Qwen3.5-35B-A3B	Alibaba	62.2
94	O3 Mini	OpenAI	61.8
95	Qwen3.5 9B	Alibaba	61.7
96	Claude Sonnet 4	Anthropic	61.1
97	Claude Opus 4	Anthropic	60.0
98	Gemini 3.5 Flash-Lite	Google	60.0
99	QwQ 32B	Alibaba	59.2
100	glm-4.7-flash	—	58.3
101	GPT-5.4	OpenAI	57.8
102	GPT-5.5	OpenAI	57.8
103	o1-2024-12-17	OpenAI	57.8
104	Qwen3 8B	Alibaba	56.1
105	Qwen3.5-4B	Alibaba	55.8
106	DeepSeek-R1-Distill-Qwen-32B	DeepSeek	55.6
107	Grok 3 Beta	xAI	55.6
108	DeepSeek R1	DeepSeek	53.3
109	GPT-5.6 Terra	OpenAI	53.3
110	qwen3-4b-instruct-2507	—	52.2
111	DeepSeek-R1-Distill-Llama-70B	DeepSeek	51.4
112	Claude Haiku 4.5	Anthropic	51.2
113	gpt-oss-20b	OpenAI	50.8
114	DeepSeek-R1-Distill-Qwen-14B	DeepSeek	50.6
115	Claude 3.7 Sonnet	Anthropic	50.0
116	Claude Opus 4.5	Anthropic	48.1
117	DeepSeek v4	DeepSeek	46.7
118	GPT-5.4 nano	OpenAI	46.7
119	Qwen3-14B	Alibaba	46.1
120	o1-mini	OpenAI	45.8
121	Qwen3 32B	Alibaba	45.0
122	GPT 4.1 mini	OpenAI	44.7
123	Gemini 2.0 Flash Thinking	Google	44.4
124	Gemini 3.1 Flash-Lite	Google	44.4
125	Qwen3-30B-A3B	Alibaba	44.2
126	DeepSeek-R1-0528	—	43.9
127	Qwen3.5 9B	Alibaba	42.8
128	GPT-5.6 Luna	OpenAI	40.0
129	GPT 4.1	OpenAI	38.3
130	DeepSeek-V3	DeepSeek	37.8
131	GPT-4.5	OpenAI	37.8
132	GPT-5.1	OpenAI	37.8
133	Claude Sonnet 4.5	Anthropic	35.6
134	Mistral Medium 3	Mistral	32.2
135	Mistral Small 3.2	Mistral	30.3
136	Magistral Small 1.0	Mistral	30.0
137	GLM-5.2	Zhipu	28.9
138	GPT-4.1 nano	OpenAI	28.9
139	Magistral Small 1.2	Mistral	28.1
140	GPT-5.4 mini	OpenAI	26.7
141	MiniMax M3	MiniMax	26.7
142	glm-4.7-flash_none	—	25.0
143	Gemma 3 27B	Google	22.5
144	Qwen3 8B	Alibaba	22.2
145	DeepSeek-R1-Distill-Qwen-1.5B	DeepSeek	21.4
146	Llama 4 Maverick Instruct	Meta	20.6
147	Qwen Plus	Alibaba	17.8
148	Gemma 3 12B	Google	16.7
149	Qwen2.5-Max	Alibaba	16.1
150	DeepSeek-V3	DeepSeek	15.8
151	Gemini 1.5 Pro	Google	14.9
152	Phi-4	Microsoft	13.8
153	Grok-2	xAI	11.5
154	Gemini 1.5 Flash	Google	10.1
155	Llama 3.1-405B	Meta	9.7
156	Claude 3.5 Sonnet	Anthropic	8.5
157	Mistral Large 2	Mistral	8.1
158	Qwen2.5-72B	Alibaba	8.1
159	Qwen3-1.7B	Alibaba	8.1
160	Llama 4 Scout Instruct	Meta	7.8
161	Gemma 3 4B	Google	7.5
162	Qwen2.5-32B	Alibaba	7.4
163	GPT-4o mini	OpenAI	6.9
164	GPT-4 Turbo	OpenAI	6.7
165	Claude 3.5 Sonnet	Anthropic	6.5
166	GPT 4o	OpenAI	6.3
167	Qwen Turbo	Alibaba	6.1
168	Mistral Small 3	Mistral	6.0
169	Llama 3.3 70B	Meta	5.1
170	Mistral Small 3.1	Mistral	4.9
171	Claude 3 Opus	Anthropic	4.7
172	Gemini 1.5 Flash 8B	Google	4.6
173	Tülu 3 70B	Ai2	4.4
174	Claude 3.5 Haiku	Anthropic	4.3
175	Llama 3-70B	Meta	4.3
176	Llama 3.1-70B	Meta	3.6
177	granite-4.0-micro	—	2.8
178	Llama 3.2 90B	Meta	2.6
179	Claude 2	Anthropic	2.5
180	Claude 3 Sonnet	Anthropic	2.5
181	Hermes 2 Theta Llama-3 70B	Nous Research	2.5
182	Qwen2.5-7B	Alibaba	2.5
183	GPT-3.5 Turbo	OpenAI	2.2
184	Claude 2.1	Anthropic	1.9
185	Llama 3-8B	Meta	1.9
186	Mistral Large	Mistral	1.9
187	Claude 3 Haiku	Anthropic	1.8
188	Llama 3.1-8B	Meta	1.7
189	Gemma 2 27B	Google	1.4
190	Gemini 1.0 Pro	Google	1.1
191	Gemma 3 1B	Google	1.1
192	GPT-4	OpenAI	1.1
193	DeepSeek LLM 67B	DeepSeek	0.8
194	granite-4.0-1b	—	0.8
195	Gemma 2 9B	Google	0.6
196	GPT-4	OpenAI	0.6
197	Llama 3.2 1B	Meta	0.6
198	granite-4.0-350m	—	0.3
199	Mistral 7B	Mistral	0.3
200	Llama 2-70B	Meta	0.0

The benchmark's own page