SimpleQA Verified
Knowledge · fixed question set
Short factual questions with one verifiable answer.
How far to trust it
Measures recall and the willingness to say 'I do not know'. A model that guesses confidently scores worse here than one that declines, which is the opposite of most benchmarks.
Measured
Models scored
83
Spread between models
18.5
standard deviation, points
Measurement noise
1.5
published stderr
Weight
0.99
share of spread that is signal
The weight is (spread² − noise²) / spread². It is computed, never chosen. Most of this benchmark's spread is real, so it still separates models.
Every model scored
the median across configurations, so one heroic run cannot lead
| Rank | Model | Lab | Score |
|---|---|---|---|
| 1 | Gemini 4 Argon | 76.0 | |
| 2 | GPT-6 Astra | OpenAI | 75.6 |
| 3 | GPT-6.1 Sol | OpenAI | 73.9 |
| 4 | Gemini 3.1 Pro Preview | 73.5 | |
| 5 | Claude Opus 5.5 | Anthropic | 72.2 |
| 6 | Claude Fable 5.1 | Anthropic | 70.8 |
| 7 | Claude Fable 5 | Anthropic | 70.7 |
| 8 | Gemini 3.8 Flash | 69.7 | |
| 9 | GPT-5.6 Sol | OpenAI | 69.7 |
| 10 | Gemini 3.7 Flash | 69.2 | |
| 11 | Gemini 3 Flash Preview | 66.8 | |
| 12 | Gemini 3.5 Flash | 66.2 | |
| 13 | Gemini 3.6 Flash | 66.2 | |
| 14 | GPT-5.5 | OpenAI | 63.0 |
| 15 | GPT-6 Sol | OpenAI | 60.7 |
| 16 | Muse Spark 1.2 | Meta | 60.3 |
| 17 | Claude Opus 5 | Anthropic | 59.9 |
| 18 | Muse Spark 1.1 | Meta | 57.8 |
| 19 | Muse Spark Open | Meta | 56.6 |
| 20 | Grok 4.7 | xAI | 56.0 |
| 21 | Qwen3.7-Max | Alibaba | 55.8 |
| 22 | Claude Opus 4.8 | Anthropic | 53.0 |
| 23 | DeepSeek V4 Pro 0813 | DeepSeek | 52.9 |
| 24 | Qwen3.6 Max Preview | Alibaba | 52.0 |
| 25 | Claude Opus 4.7 | Anthropic | 51.7 |
| 26 | Kimi K3 | Kimi | 50.6 |
| 27 | GPT 5 | OpenAI | 50.1 |
| 28 | o3 | OpenAI | 49.4 |
| 29 | Grok 4.6 | xAI | 49.1 |
| 30 | Qwen3 Max | Alibaba | 48.8 |
| 31 | Grok 4.5 | xAI | 48.3 |
| 32 | GPT 5.1 | OpenAI | 48.0 |
| 33 | Qwen3.8 Max (0902) | Alibaba | 47.3 |
| 34 | Claude Opus 4.6 | Anthropic | 47.0 |
| 35 | DeepSeek v4 | DeepSeek | 47.0 |
| 36 | Claude Sonnet 5.5 | Anthropic | 46.5 |
| 37 | GPT-5.4 Pro | OpenAI | 46.3 |
| 38 | Qwen3.8 Max | Alibaba | 45.8 |
| 39 | Claude Opus 4.5 | Anthropic | 45.7 |
| 40 | GPT-5.4 | OpenAI | 45.1 |
| 41 | Qwen3.6 Plus | Alibaba | 44.1 |
| 42 | GPT-5.6 Terra | OpenAI | 43.2 |
| 43 | DeepSeek V4.1 Flash | DeepSeek | 42.9 |
| 44 | GPT-6 Luna | OpenAI | 41.4 |
| 45 | o1-2024-12-17 | OpenAI | 41.1 |
| 46 | GLM-5.3 | Zhipu | 41.0 |
| 47 | GPT-5.6 Luna | OpenAI | 41.0 |
| 48 | Qwen3-235B-A22B | Alibaba | 40.4 |
| 49 | Inkling | Thinking Machines | 40.3 |
| 50 | Kimi K2.7 Code | Kimi | 36.5 |
| 51 | Kimi K2.6 | Kimi | 34.9 |
| 52 | Kimi K2.5 | Kimi | 34.3 |
| 53 | GLM-5.2 | Zhipu | 34.2 |
| 54 | Claude Sonnet 4.6 | Anthropic | 34.1 |
| 55 | GLM-5.1 | Zhipu | 34.0 |
| 56 | DeepSeek V4 Flash 0731 | DeepSeek | 33.6 |
| 57 | GPT 5.2 | OpenAI | 33.5 |
| 58 | Claude Sonnet 5 | Anthropic | 33.3 |
| 59 | Grok 4.3 Beta | xAI | 33.2 |
| 60 | GLM-4.7 | Zhipu | 32.2 |
| 61 | GPT 4.1 | OpenAI | 31.1 |
| 62 | Claude Sonnet 4.5 | Anthropic | 30.7 |
| 63 | Grok 4.20 | xAI | 30.2 |
| 64 | GPT-5.4 mini | OpenAI | 29.4 |
| 65 | GPT 4o | OpenAI | 26.0 |
| 66 | Qwen3.5 Plus | Alibaba | 25.4 |
| 67 | Claude Haiku 5.5 | Anthropic | 23.8 |
| 68 | Claude Sonnet 4.5 | Anthropic | 23.7 |
| 69 | GPT 5 mini | OpenAI | 21.6 |
| 70 | Qwen3.5 Flash | Alibaba | 20.3 |
| 71 | Mistral Large 4 | Mistral | 20.0 |
| 72 | o4-mini | OpenAI | 19.2 |
| 73 | Inkling Small | Thinking Machines | 19.1 |
| 74 | Qwen3.6 Flash | Alibaba | 15.9 |
| 75 | O3 Mini | OpenAI | 15.3 |
| 76 | Claude Haiku 4.5 | Anthropic | 12.9 |
| 77 | GPT 4.1 mini | OpenAI | 12.7 |
| 78 | Claude 3 Opus | Anthropic | 12.6 |
| 79 | GPT-5.4 nano | OpenAI | 11.7 |
| 80 | GPT 5 nano | OpenAI | 11.7 |
| 81 | Gemma 4 31B IT | 10.4 | |
| 82 | GPT-4o mini | OpenAI | 8.3 |
| 83 | GPT-4.1 nano | OpenAI | 6.0 |