← Benchmarks
SWE-Bench Verified
A verified subset of 500 software engineering problems from real GitHub issues, validated by human annotators for evaluating language models' ability to resolve real-world coding issues by generating patches for Python codebases.
id swe-bench-verified · max 1 · 116 models reported
| # | Model | Score |
|---|---|---|
| 1 | Claude Fable 5 Anthropic | 0.95 |
| 2 | Claude Mythos Preview Anthropic | 0.94 |
| 3 | Claude Opus 4.8 Anthropic | 0.89 |
| 4 | Claude Opus 4.7 Anthropic | 0.88 |
| 5 | Claude Sonnet 5 Anthropic | 0.85 |
| 6 | Claude Opus 4.5 Anthropic | 0.81 |
| 7 | Claude Opus 4.6 Anthropic | 0.81 |
| 8 | DeepSeek-V4-Pro-Max DeepSeek · open | 0.81 |
| 9 | Gemini 3.1 Pro Google | 0.81 |
| 10 | MiniMax M3 MiniMax · open | 0.81 |
| 11 | Qwen3.7 Max Alibaba Cloud / Qwen Team | 0.80 |
| 12 | Inkling-Small Thinking Machines Lab · open | 0.80 |
| 13 | Kimi K2.6 Moonshot AI · open | 0.80 |
| 14 | MiniMax M2.5 MiniMax · open | 0.80 |
| 15 | GPT-5.2 OpenAI | 0.80 |
| 16 | Claude Sonnet 4.6 Anthropic | 0.80 |
| 17 | DeepSeek-V4-Flash-Max DeepSeek · open | 0.79 |
| 18 | MiMo-V2.5-Pro Xiaomi · open | 0.79 |
| 19 | Qwen3.6 Plus Alibaba Cloud / Qwen Team | 0.79 |
| 20 | DeepSeek-V4-Flash-0423 DeepSeek · open | 0.79 |
| 21 | Gemini 3 Flash Google | 0.78 |
| 22 | Hy3 Tencent · open | 0.78 |
| 23 | MiMo-V2-Pro Xiaomi | 0.78 |
| 24 | GLM-5 Zhipu AI · open | 0.78 |
| 25 | Qwen3.7-Plus Alibaba Cloud / Qwen Team | 0.78 |
| 26 | Inkling Thinking Machines Lab · open | 0.78 |
| 27 | Mistral Medium 3.5 Mistral AI · open | 0.78 |
| 28 | Muse Spark Meta | 0.77 |
| 29 | Qwen3.6-27B Alibaba Cloud / Qwen Team · open | 0.77 |
| 30 | Kimi K2.5 Moonshot AI · open | 0.77 |
| 31 | Seed 2.0 Pro ByteDance | 0.77 |
| 32 | Qwen3.5-397B-A17B Alibaba Cloud / Qwen Team · open | 0.76 |
| 33 | GPT-5.1 OpenAI | 0.76 |
| 34 | GPT-5.1 Instant OpenAI | 0.76 |
| 35 | GPT-5.1 Thinking OpenAI | 0.76 |
| 36 | Gemini 3 Pro Google | 0.76 |
| 37 | Muse Glimmer-30B Meta · open | 0.76 |
| 38 | Qwen3 Max Thinking Alibaba Cloud / Qwen Team | 0.75 |
| 39 | GPT-5 OpenAI | 0.75 |
| 40 | MiMo-V2-Omni Xiaomi | 0.75 |
| 41 | Claude Opus 4.1 Anthropic | 0.74 |
| 42 | GPT-5 Codex OpenAI | 0.74 |
| 43 | Step-3.5-Flash StepFun · open | 0.74 |
| 44 | GLM-4.7 Zhipu AI · open | 0.74 |
| 45 | GPT-5.1 Codex OpenAI | 0.74 |
| 46 | MAI-Thinking-1 Microsoft | 0.73 |
| 47 | Seed 2.0 Lite ByteDance | 0.73 |
| 48 | MiMo-V2-Flash Xiaomi · open | 0.73 |
| 49 | Qwen3.6-35B-A3B Alibaba Cloud / Qwen Team · open | 0.73 |
| 50 | Claude Haiku 4.5 Anthropic | 0.73 |
| 51 | DeepSeek-V3.2 DeepSeek · open | 0.73 |
| 52 | DeepSeek-V3.2 (Thinking) DeepSeek · open | 0.73 |
| 53 | DeepSeek-V3.2-Speciale DeepSeek · open | 0.73 |
| 54 | Seed 1.8 ByteDance | 0.73 |
| 55 | Claude Sonnet 4 Anthropic | 0.73 |
| 56 | MAI-Code-1.1-Flash Microsoft | 0.73 |
| 57 | Claude Opus 4 Anthropic | 0.72 |
| 58 | Qwen3.5-27B Alibaba Cloud / Qwen Team · open | 0.72 |
| 59 | Qwen3.5-122B-A10B Alibaba Cloud / Qwen Team · open | 0.72 |
| 60 | MAI-Code-1-Flash Microsoft | 0.72 |
| 61 | Kimi K2-Thinking-0905 Moonshot AI · open | 0.71 |
| 62 | Laguna XS 2.1 Poolside · open | 0.71 |
| 63 | Grok Code Fast 1 xAI | 0.71 |
| 64 | Nemotron 3 Ultra (550B A55B) NVIDIA · open | 0.71 |
| 65 | Solar Pro 4 Upstage | 0.71 |
| 66 | Claude 3.7 Sonnet Anthropic | 0.70 |
| 67 | LongCat-Flash-Thinking-2601 Meituan · open | 0.70 |
| 68 | Nova 2 Pro Amazon | 0.70 |
| 69 | Qwen3 Max Alibaba Cloud / Qwen Team | 0.70 |
| 70 | Qwen3-Coder 480B A35B Instruct Alibaba Cloud / Qwen Team · open | 0.70 |
| 71 | MiniMax M2 MiniMax · open | 0.69 |
| 72 | Qwen3.5-35B-A3B Alibaba Cloud / Qwen Team · open | 0.69 |
| 73 | o3 OpenAI | 0.69 |
| 74 | o4-mini OpenAI | 0.68 |
| 75 | GLM-4.6 Zhipu AI · open | 0.68 |
| 76 | DeepSeek-V3.2-Exp DeepSeek · open | 0.68 |
| 77 | North Mini Code 1.0 Cohere · open | 0.68 |
| 78 | Gemini 2.5 Pro Preview 06-05 Google | 0.67 |
| 79 | MiniMax M2.1 MiniMax · open | 0.67 |
| 80 | DeepSeek-V3.1 DeepSeek · open | 0.66 |
| 81 | Kimi K2-Instruct-0905 Moonshot AI · open | 0.66 |
| 82 | Nova 2 Lite Amazon | 0.65 |
| 83 | GLM-4.5 Zhipu AI · open | 0.64 |
| 84 | Gemini 2.5 Pro Google | 0.63 |
| 85 | Devstral Medium Mistral AI | 0.62 |
| 86 | Gemini 2.5 Flash Google | 0.60 |
| 87 | LongCat-Flash-Chat Meituan · open | 0.60 |
| 88 | LongCat-Flash-Thinking Meituan · open | 0.59 |
| 89 | GLM-4.7-Flash Zhipu AI · open | 0.59 |
| 90 | GLM-4.5-Air Zhipu AI · open | 0.58 |
| 91 | IBM Granite 4.2 30B IBM · open | 0.57 |
| 92 | MiniMax M1 80K MiniMax · open | 0.56 |
| 93 | MiniMax M1 40K MiniMax · open | 0.56 |
| 94 | GPT-4.1 OpenAI | 0.55 |
| 95 | LongCat-Flash-Lite Meituan · open | 0.54 |
| 96 | Nemotron 3 Super (120B A12B) NVIDIA · open | 0.54 |
| 97 | Devstral Small 1.1 Mistral AI · open | 0.54 |
| 98 | Nemotron 3.5 Lightning (30B A3B) NVIDIA · open | 0.52 |
| 99 | o3-mini OpenAI | 0.49 |
| 100 | Claude 3.5 Sonnet Anthropic | 0.49 |