MMLU-Pro

A more robust and challenging multi-task language understanding benchmark that extends MMLU by expanding multiple-choice options from 4 to 10, eliminating trivial questions, and focusing on reasoning-intensive tasks. Features over 12,000 curated questions across 14 domains and causes a 16-33% accuracy drop compared to original MMLU.

id mmlu-pro · max 1 · 142 models reported

#	Model	Score
1	Sakana Namazu Sakana AI	0.90
2	Qwen3.7 Max Alibaba Cloud / Qwen Team	0.90
3	Qwen3.6 Plus Alibaba Cloud / Qwen Team	0.89
4	Qwen3.7-Plus Alibaba Cloud / Qwen Team	0.89
5	MiniMax M2.1 MiniMax · open	0.88
6	Qwen3.5-397B-A17B Alibaba Cloud / Qwen Team · open	0.88
7	DeepSeek-V4-Pro-Max DeepSeek · open	0.88
8	Kimi K2.5 Moonshot AI · open	0.87
9	ERNIE 5.0 Baidu	0.87
10	Nemotron 3 Ultra (550B A55B) NVIDIA · open	0.87
11	Qwen3.5-122B-A10B Alibaba Cloud / Qwen Team · open	0.87
12	DeepSeek-V4-Flash-0423 DeepSeek · open	0.86
13	Solar Pro 4 Upstage	0.86
14	DeepSeek-V4-Flash-Max DeepSeek · open	0.86
15	Qwen3.6-27B Alibaba Cloud / Qwen Team · open	0.86
16	Qwen3.5-27B Alibaba Cloud / Qwen Team · open	0.86
17	Qwen3 Max Thinking Alibaba Cloud / Qwen Team	0.86
18	Qwen3.5-35B-A3B Alibaba Cloud / Qwen Team · open	0.85
19	Gemma 4 31B Google · open	0.85
20	Qwen3.6-35B-A3B Alibaba Cloud / Qwen Team · open	0.85
21	DeepSeek-R1-0528 DeepSeek · open	0.85
22	DeepSeek-V3.2 DeepSeek · open	0.85
23	DeepSeek-V3.2 (Thinking) DeepSeek · open	0.85
24	DeepSeek-V3.2-Exp DeepSeek · open	0.85
25	MAI-Thinking-1 Microsoft	0.85
26	MiMo-V2-Flash Xiaomi · open	0.85
27	Seed 1.8 ByteDance	0.85
28	GLM-4.5 Zhipu AI · open	0.85
29	Kimi K2-Thinking-0905 Moonshot AI · open	0.85
30	Qwen3-235B-A22B-Thinking-2507 Alibaba Cloud / Qwen Team · open	0.84
31	GLM-4.7 Zhipu AI · open	0.84
32	K-EXAONE-236B-A23B LG AI Research	0.84
33	Qwen3 VL 235B A22B Thinking Alibaba Cloud / Qwen Team · open	0.84
34	Nemotron 3 Super (120B A12B) NVIDIA · open	0.84
35	DeepSeek-V3.1 DeepSeek · open	0.84
36	Seed 2.0 Mini ByteDance	0.84
37	EXAONE 4.5 33B LG AI Research	0.83
38	Qwen3-235B-A22B-Instruct-2507 Alibaba Cloud / Qwen Team · open	0.83
39	Qwen3-Next-80B-A3B-Thinking Alibaba Cloud / Qwen Team · open	0.83
40	LongCat-Flash-Chat Meituan · open	0.83
41	Gemma 4 26B-A4B Google · open	0.83
42	LongCat-Flash-Thinking Meituan · open	0.83
43	Kimi K2 0905 Moonshot AI	0.82
44	Qwen3.5-9B Alibaba Cloud / Qwen Team · open	0.82
45	Qwen3 VL 32B Thinking Alibaba Cloud / Qwen Team · open	0.82
46	MiniMax M2 MiniMax · open	0.82
47	Nemotron 3.5 Lightning (30B A3B) NVIDIA · open	0.82
48	Qwen3 VL 235B A22B Instruct Alibaba Cloud / Qwen Team · open	0.82
49	Sarvam-105B Sarvam AI · open	0.82
50	Nova 2 Pro Amazon	0.82
51	GLM-4.5-Air Zhipu AI · open	0.81
52	DeepSeek-V3 0324 DeepSeek · open	0.81
53	Kimi K2 Instruct Moonshot AI · open	0.81
54	Kimi K2-Instruct-0905 Moonshot AI · open	0.81
55	MiniMax M1 80K MiniMax · open	0.81
56	Nova 2 Lite Amazon	0.81
57	GPT OSS 120B High OpenAI · open	0.81
58	Nova 2 Omni Amazon	0.81
59	MiniMax M1 40K MiniMax · open	0.81
60	Qwen3-Next-80B-A3B-Instruct Alibaba Cloud / Qwen Team · open	0.81
61	Llama 4 Maverick Meta · open	0.81
62	Qwen3 VL 30B A3B Thinking Alibaba Cloud / Qwen Team · open	0.81
63	Sarvam-30B Sarvam AI · open	0.80
64	Qwen3.5-4B Alibaba Cloud / Qwen Team · open	0.79
65	Qwen3 VL 32B Instruct Alibaba Cloud / Qwen Team · open	0.79
66	Nemotron 3 Nano (30B A3B) NVIDIA · open	0.78
67	LongCat-Flash-Lite Meituan · open	0.78
68	Mistral Small 4 Mistral AI · open	0.78
69	Qwen3 VL 30B A3B Instruct Alibaba Cloud / Qwen Team · open	0.78
70	Claude 3.5 Sonnet Anthropic	0.78
71	DiffusionGemma 26B-A4B Google · open	0.78
72	IBM Granite 4.2 30B IBM · open	0.78
73	Qwen3 VL 8B Thinking Alibaba Cloud / Qwen Team · open	0.77
74	Gemma 4 12B Google · open	0.77
75	Gemini 2.0 Flash Google	0.76
76	Claude 3.5 Sonnet Anthropic	0.76
77	Phi 4 Reasoning Plus Microsoft · open	0.76
78	DeepSeek-V3 DeepSeek · open	0.76
79	Gemini 1.5 Pro Google	0.76
80	Grok-2 xAI	0.76
81	GPT-4o OpenAI	0.75
82	Llama 4 Scout Meta · open	0.74
83	Phi 4 Reasoning Microsoft · open	0.74
84	IBM Granite 4.2 8B IBM · open	0.74
85	Qwen3 VL 4B Thinking Alibaba Cloud / Qwen Team · open	0.74
86	Llama 3.1 405B Instruct Meta · open	0.73
87	GPT-4o OpenAI	0.73
88	Grok-2 mini xAI	0.72
89	Gemini 2.0 Flash-Lite Google	0.72
90	Qwen3 VL 8B Instruct Alibaba Cloud / Qwen Team · open	0.72
91	Qwen2.5 72B Instruct Alibaba Cloud / Qwen Team · open	0.71
92	Phi 4 Microsoft · open	0.70
93	Gemma 4 E4B Google · open	0.69
94	Kimi K2 Base Moonshot AI · open	0.69
95	Mistral Small 3.2 24B Instruct Mistral AI · open	0.69
96	Qwen2.5 32B Instruct Alibaba Cloud / Qwen Team · open	0.69
97	Llama 3.3 70B Instruct Meta · open	0.69
98	Qwen2.5 VL 32B Instruct Alibaba Cloud / Qwen Team · open	0.69
99	Claude 3 Opus Anthropic	0.69
100	MiMo-V2.5-Pro Xiaomi · open	0.69