HumanEval

A benchmark that measures functional correctness for synthesizing programs from docstrings, consisting of 164 original programming problems assessing language comprehension, algorithms, and simple mathematics

id humaneval · max 1 · 66 models reported

#	Model	Score
1	MiniCPM-SALA OpenBMB · open	0.95
2	Kimi K2 0905 Moonshot AI	0.94
3	Claude 3.5 Sonnet Anthropic	0.94
4	GPT-5 OpenAI	0.93
5	Kimi K2 Instruct Moonshot AI · open	0.93
6	Qwen2.5-Coder 32B Instruct Alibaba Cloud / Qwen Team · open	0.93
7	o1-mini OpenAI	0.92
8	Sarvam-30B Sarvam AI · open	0.92
9	Claude 3.5 Sonnet Anthropic	0.92
10	Mistral Large 2 Mistral AI · open	0.92
11	Qwen2.5 VL 32B Instruct Alibaba Cloud / Qwen Team · open	0.92
12	GPT-4o OpenAI	0.90
13	Granite 3.3 8B Base IBM · open	0.90
14	Granite 3.3 8B Instruct IBM · open	0.90
15	Gemini Diffusion Google	0.90
16	DeepSeek-V2.5 DeepSeek · open	0.89
17	Llama 3.1 405B Instruct Meta · open	0.89
18	Nova Pro Amazon	0.89
19	LongCat-Flash-Chat Meituan · open	0.88
20	Mistral Small 3.1 24B Instruct Mistral AI · open	0.88
21	Grok-2 xAI	0.88
22	Llama 3.3 70B Instruct Meta · open	0.88
23	Qwen2.5 32B Instruct Alibaba Cloud / Qwen Team · open	0.88
24	Qwen2.5-Coder 7B Instruct Alibaba Cloud / Qwen Team · open	0.88
25	Claude 3.5 Haiku Anthropic	0.88
26	o1 OpenAI	0.88
27	GPT-4.5 OpenAI	0.88
28	Gemma 3 27B Google · open	0.88
29	GPT-4o mini OpenAI	0.87
30	GPT-4 Turbo OpenAI	0.87
31	Qwen2.5 72B Instruct Alibaba Cloud / Qwen Team · open	0.87
32	Qwen2 72B Instruct Alibaba Cloud / Qwen Team · open	0.86
33	Grok-2 mini xAI	0.86
34	Gemma 3 12B Google · open	0.85
35	Nova Lite Amazon	0.85
36	Claude 3 Opus Anthropic	0.85
37	Mistral Small 3 24B Instruct Mistral AI · open	0.85
38	Qwen2.5 7B Instruct Alibaba Cloud / Qwen Team · open	0.85
39	Gemini 1.5 Pro Google	0.84
40	Qwen2.5 14B Instruct Alibaba Cloud / Qwen Team · open	0.83
41	Phi 4 Microsoft · open	0.83
42	IBM Granite 4.0 Tiny Preview IBM · open	0.82
43	Codestral-22B Mistral AI · open	0.81
44	Nova Micro Amazon	0.81
45	Llama 3.1 70B Instruct Meta · open	0.81
46	Qwen2 7B Instruct Alibaba Cloud / Qwen Team · open	0.80
47	Qwen2.5-Omni-7B Alibaba Cloud / Qwen Team · open	0.79
48	Claude 3 Haiku Anthropic	0.76
49	Gemma 3n E4B Instructed Google	0.75
50	Gemma 3n E4B Instructed LiteRT Preview Google · open	0.75
51	Gemini 1.5 Flash Google	0.74
52	Grok-1.5 xAI	0.74
53	Claude 3 Sonnet Anthropic	0.73
54	Llama 3.1 8B Instruct Meta · open	0.73
55	Pixtral-12B Mistral AI · open	0.72
56	Gemma 3 4B Google · open	0.71
57	Phi-3.5-MoE-instruct Microsoft · open	0.71
58	GPT-3.5 Turbo OpenAI	0.68
59	GPT-4 OpenAI	0.67
60	Gemma 3n E2B Instructed Google	0.67
61	Gemma 3n E2B Instructed LiteRT (Preview) Google · open	0.67
62	Phi-3.5-mini-instruct Microsoft · open	0.63
63	Gemma 2 27B Google · open	0.52
64	Gemma 3 1B Google · open	0.41
65	Gemma 2 9B Google · open	0.40
66	Ministral 8B Instruct Mistral AI · open	0.35