Pencil Puzzle Bench
A Benchmark for Multi-Step Verifiable Reasoning
Can AI solve pencil puzzles? We evaluated 51 frontier models on a selection of 300 puzzles from our 62k puzzle dataset, spanning 20 types.
62,231
Dataset Puzzles
20
Puzzle Types
51
Models Tested
17k
Eval Runs
Model Leaderboard
| # ▼ | Model | Direct Ask | Agentic | Cost/Agentic |
|---|---|---|---|---|
| 1 | claude-fable-5@highAnthropic | 50.0% | 97.6% | $19.6630 |
| 2 | gpt-5.5@xhighOpenAI | -- | 83.3% | $3.0703 |
| 3 | gpt-5.4@xhighOpenAI | -- | 70.2% | $8.0759 |
| 4 | gpt-5.2@xhighOpenAI | 27.0% | 56.0% | $9.7432 |
| 5 | claude-opus-4-7@thinkingAnthropic | -- | 50.0% | $9.6427 |
| 6 | gemini-3.5-flash@highGoogle | 13.0% | 41.9% | $24.8795 |
| 7 | qwen3.7-maxQwen | 7.3% | 40.0% | $8.8604 |
| 8 | gpt-5.2@highOpenAI | 20.7% | 36.7% | $7.3048 |
| 9 | claude-opus-4-6-1mAnthropic | 0.0% | 36.7% | $14.1105 |
| 10 | claude-opus-4-6@thinkingAnthropic | 27.3% | 33.3% | $6.2425 |
| 11 | gemini-3.1-proGoogle | 20.0% | 33.3% | $3.6678 |
| 12 | claude-opus-4-6Anthropic | 0.3% | 30.0% | $10.8893 |
| 13 | glm-5.2Zhipu | 16.0% | 26.7% | $20.6620 |
| 14 | claude-sonnet-4-6@thinkingAnthropic | 10.3% | 26.7% | $3.9361 |
| 15 | gpt-5.2-proOpenAI | 9.7% | 26.7% | $41.5243 |
| 16 | gpt-5.2@mediumOpenAI | 9.3% | 23.3% | $2.8149 |
| 17 | claude-opus-4-6@maxAnthropic | 0.3% | 23.3% | $10.8699 |
| 18 | claude-sonnet-4-6-1mAnthropic | 0.3% | 23.3% | $169.2700 |
| 19 | kimi-k2.6Moonshot | 11.7% | 20.0% | $6.2137 |
| 20 | kimi-k2.7-codeMoonshot | 6.7% | 16.7% | $3.9212 |
| 21 | qwen3.7-plusQwen | 5.3% | 16.7% | $10.3589 |
| 22 | gemini-3-pro@highGoogle | 3.3% | 16.7% | $3.0609 |
| 23 | claude-sonnet-4-6Anthropic | 0.3% | 16.7% | $8.9114 |
| 24 | gemini-3-proGoogle | 4.3% | 13.3% | $2.4814 |
| 25 | gemini-3-pro@minimalGoogle | 4.0% | 10.0% | $3.3548 |
| 26 | gpt-5.2@lowOpenAI | 2.3% | 10.0% | $0.5968 |
| 27 | qwen3.6-plusQwen | 0.3% | 10.0% | $0.0000 |
| 28 | gpt-5.1@mediumOpenAI | 7.7% | 6.7% | $0.8125 |
| 29 | minimax-m3Minimax | 3.3% | 7.1% | $10.9871 |
| 30 | claude-opus-4-5@thinkingAnthropic | 6.0% | 6.7% | $4.7130 |
| 31 | gemini-3-flash@minimalGoogle | 4.7% | 6.7% | $0.4156 |
| 32 | gemini-3-flash@highGoogle | 3.0% | 6.7% | $0.3958 |
| 33 | grok-4.20-reasoningxAI | 0.3% | 6.7% | $1.1939 |
| 34 | gpt-5@mediumOpenAI | 6.0% | 3.3% | $11.2062 |
| 35 | kimi-k2.5Moonshot | 6.0% | 3.3% | $1.7154 |
| 36 | grok-4-1-fastxAI | 5.7% | 3.3% | $0.4391 |
| 37 | grok-4-1-fast-reasoningxAI | 5.3% | 0.0% | $0.4880 |
| 38 | deepseek-v4-proDeepSeek | 4.0% | 0.0% | $3.5881 |
| 39 | grok-4.3@xhighxAI | 3.3% | 3.3% | $58.5513 |
| 40 | o3OpenAI | 3.0% | 3.3% | $2.4470 |
| 41 | nemotron-3-ultra-550b-a55bOther | 3.3% | 0.0% | $8.7289 |
| 42 | minimax-m2.5Minimax | 0.7% | 3.3% | $1.9467 |
| 43 | claude-opus-4-5-highAnthropic | 0.3% | 3.3% | $8.8238 |
| 44 | claude-sonnet-4-5Anthropic | 0.0% | 3.3% | $12.1957 |
| 45 | deepseek-v3.2-specialeDeepSeek | 2.3% | -- | -- |
| 46 | claude-sonnet-4-5@thinkingAnthropic | 2.3% | 0.0% | $7.4948 |
| 47 | deepseek-v3.2DeepSeek | 2.0% | 0.0% | $1.5830 |
| 48 | grok-4.3xAI | 2.0% | 0.0% | $12.7441 |
| 49 | kimi-k2-thinkingMoonshot | 1.3% | 0.0% | $1.0019 |
| 50 | mimo-v2-proXiaomi | 1.0% | 0.0% | $9.2958 |
| 51 | o1OpenAI | 0.7% | 0.0% | $4.0281 |
| 52 | minimax-m2.7Minimax | 0.7% | 0.0% | $3.7122 |
| 53 | qwen3.5-397b-a17bQwen | 0.7% | 0.0% | $7.3052 |
| 54 | glm-5Zhipu | 0.7% | 0.0% | $8.4639 |
| 55 | gemini-2.5-proGoogle | 0.3% | 0.0% | $1.1731 |
| 56 | gpt-5.2OpenAI | 0.3% | 0.0% | $0.6118 |
| 57 | gemma-4-31b-itOther | 0.3% | 0.0% | $0.0145 |
| 58 | minimax-m2.1Minimax | 0.3% | 0.0% | $1.6709 |
| 59 | gpt-oss-120bOpenAI | 0.3% | -- | -- |
| 60 | qwen3-235b-a22b-thinking-2507Qwen | 0.3% | 0.0% | $0.7058 |
| 61 | qwen3-next-80b-a3b-thinkingQwen | 0.3% | 0.0% | $2.6243 |
| 62 | qwen3-vl-235b-a22b-thinkingQwen | 0.3% | -- | -- |
| 63 | mimo-v2-flashXiaomi | 0.3% | 0.0% | $0.7595 |
| 64 | glm-4.7Zhipu | 0.3% | 0.0% | $1.7271 |
| 65 | grok-code-fast-1xAI | 0.3% | 0.0% | $1.0183 |
| 66 | gemini-3.5-flash@lowGoogle | -- | 0.0% | $73.5232 |
| 67 | gpt-3.5-turboOpenAI | 0.0% | 0.0% | $0.0000 |
| 68 | gpt-4.1OpenAI | 0.0% | 0.0% | $220.4705 |
| 69 | gpt-4oOpenAI | 0.0% | 0.0% | $55.5660 |
| 70 | devstral-2512Mistral | 0.0% | 0.0% | $0.1113 |
| 71 | mistral-large-2512Mistral | 0.0% | 0.0% | $38.6499 |
| 72 | mistral-small-2603Mistral | 0.0% | 0.0% | $10.0508 |
| 73 | qwen3-coderQwen | 0.0% | 0.0% | $0.6826 |