Pinyin Letter Count

A benchmark to evaluate a model's ability to count how many times a specific letter appears in the Pinyin representation of a Chinese sentence.

Run Benchmark
Questions

20

Leaderboard Entries

27

Best Score

60/100

Leaderboard
Rank Model Score Run Date Actions
Gemma 2B (LMStudio)
1500 MB
60
349.0ms median
 2026-02-28 15:34:16 View Details
2 Gemma 3 12B (LMStudio)
8100 MB
50
1626.0ms median
 2026-02-28 15:50:17 View Details
3 GPT-5.4 mini
40
740.0ms median
cost warning
2026-03-17 18:52:23 View Details
4 Gemma 4 12B (LMStudio)
7560 MB
35
1446.0ms median
 2026-06-03 18:45:20 View Details
5 Phi-4 (LMStudio)
9100 MB
35
1092.0ms median
 2026-03-03 00:49:36 View Details
6 Claude Haiku 4.5
35
671.5ms median
 2026-02-28 19:43:34 View Details
7 SmolLM2 1.7B (LMStudio)
1100 MB
35
220.5ms median
 2026-02-28 19:18:46 View Details
8 GPT-5 mini
35
1135.0ms median
 2026-02-26 01:23:39 View Details
9 Gemma 2 9B (LMStudio)
5800 MB
30
810.0ms median
 2026-02-28 14:59:17 View Details
10 Gemma 2 2B (LMStudio)
1500 MB
30
692.5ms median
 2026-02-28 03:21:28 View Details
11 Qwen3.5 9B (LMStudio)
6600 MB
25
2369.5ms median
 2026-03-03 19:53:39 View Details
12 Llama 3 8B (LMStudio)
4900 MB
25
533.0ms median
 2026-02-28 16:45:15 View Details
13 Granite 3.2 8B (LMStudio)
4900 MB
25
772.5ms median
 2026-02-28 16:17:13 View Details
14 GPT-5.4 nano
20
682.5ms median
 2026-03-17 19:15:18 View Details
15 Qwen3 VL 8B (LMStudio)
5000 MB
20
3396.5ms median
 2026-02-28 18:35:24 View Details
16 OLMo 3 7B (LMStudio)
4300 MB
20
466.5ms median
 2026-02-28 17:36:07 View Details
17 Gemma 4 E4B IT (LMStudio)
3200 MB
15
1059.0ms median
 2026-04-02 20:57:10 View Details
18 Ministral 8B (LMStudio)
4900 MB
15
567.0ms median
 2026-02-28 17:21:45 View Details
19 Llama 3.1 8B (LMStudio)
4900 MB
15
526.0ms median
 2026-02-28 17:00:38 View Details
20 Llama 2 7B (LMStudio)
4900 MB
15
678.5ms median
 2026-02-28 03:04:01 View Details
21 GPT-5 nano
15
1066.0ms median
 2026-02-26 00:56:32 View Details
22 Llama 3.2 1B (LMStudio)
1300 MB
10
203.0ms median
 2026-02-28 17:12:30 View Details
23 Qwen3.5 2B (LMStudio)
2700 MB
5
547.0ms median
 2026-03-03 19:20:17 View Details
24 Qwen3 4B (LMStudio)
2800 MB
5
414.0ms median
 2026-02-28 18:12:27 View Details
25 Qwen3 1.7B (LMStudio)
1100 MB
5
220.5ms median
 2026-02-28 18:04:43 View Details
26 Qwen3.5 4B (LMStudio)
3400 MB
0
1344.0ms median
 2026-03-03 21:00:42 View Details
27 Phi-3.5 Mini (LMStudio)
2500 MB
0
766.0ms median
 2026-02-28 17:48:31 View Details
Questions

Question
Count how many times the letter 'U' appears in the Pinyin representation of the following Chinese sentence: 我喜欢学习中文
Question payload
{
  "question_text": "Count how many times the letter 'U' appears in the Pinyin representation of the following Chinese sentence: 我喜欢学习中文",
  "answer_type": "numeric",
  "correct_answer": 2,
  "category": "pinyin_letter_count",
  "difficulty": "medium",
  "tags": [
    "chinese",
    "pinyin",
    "letter_count"
  ],
  "evaluation_criteria": {
    "exact_match": true,
    "case_sensitive": false,
    "contains": false,
    "required_fields": [],
    "tolerance": 0.0,
    "alternatives": []
  }
}

Question
Count how many times the letter 'E' appears in the Pinyin representation of the following Chinese sentence: 北京是中国的首都
Question payload
{
  "question_text": "Count how many times the letter 'E' appears in the Pinyin representation of the following Chinese sentence: 北京是中国的首都",
  "answer_type": "numeric",
  "correct_answer": 2,
  "category": "pinyin_letter_count",
  "difficulty": "medium",
  "tags": [
    "chinese",
    "pinyin",
    "letter_count"
  ],
  "evaluation_criteria": {
    "exact_match": true,
    "case_sensitive": false,
    "contains": false,
    "required_fields": [],
    "tolerance": 0.0,
    "alternatives": []
  }
}

Question
Count how many times the letter 'J' appears in the Pinyin representation of the following Chinese sentence: 你叫什么名字
Question payload
{
  "question_text": "Count how many times the letter 'J' appears in the Pinyin representation of the following Chinese sentence: 你叫什么名字",
  "answer_type": "numeric",
  "correct_answer": 1,
  "category": "pinyin_letter_count",
  "difficulty": "medium",
  "tags": [
    "chinese",
    "pinyin",
    "letter_count"
  ],
  "evaluation_criteria": {
    "exact_match": true,
    "case_sensitive": false,
    "contains": false,
    "required_fields": [],
    "tolerance": 0.0,
    "alternatives": []
  }
}

Question
Count how many times the letter 'U' appears in the Pinyin representation of the following Chinese sentence: 我想喝水
Question payload
{
  "question_text": "Count how many times the letter 'U' appears in the Pinyin representation of the following Chinese sentence: 我想喝水",
  "answer_type": "numeric",
  "correct_answer": 1,
  "category": "pinyin_letter_count",
  "difficulty": "medium",
  "tags": [
    "chinese",
    "pinyin",
    "letter_count"
  ],
  "evaluation_criteria": {
    "exact_match": true,
    "case_sensitive": false,
    "contains": false,
    "required_fields": [],
    "tolerance": 0.0,
    "alternatives": []
  }
}

Question
Count how many times the letter 'H' appears in the Pinyin representation of the following Chinese sentence: 他在看书
Question payload
{
  "question_text": "Count how many times the letter 'H' appears in the Pinyin representation of the following Chinese sentence: 他在看书",
  "answer_type": "numeric",
  "correct_answer": 1,
  "category": "pinyin_letter_count",
  "difficulty": "medium",
  "tags": [
    "chinese",
    "pinyin",
    "letter_count"
  ],
  "evaluation_criteria": {
    "exact_match": true,
    "case_sensitive": false,
    "contains": false,
    "required_fields": [],
    "tolerance": 0.0,
    "alternatives": []
  }
}

Question
Count how many times the letter 'T' appears in the Pinyin representation of the following Chinese sentence: 今天很热
Question payload
{
  "question_text": "Count how many times the letter 'T' appears in the Pinyin representation of the following Chinese sentence: 今天很热",
  "answer_type": "numeric",
  "correct_answer": 1,
  "category": "pinyin_letter_count",
  "difficulty": "medium",
  "tags": [
    "chinese",
    "pinyin",
    "letter_count"
  ],
  "evaluation_criteria": {
    "exact_match": true,
    "case_sensitive": false,
    "contains": false,
    "required_fields": [],
    "tolerance": 0.0,
    "alternatives": []
  }
}

Question
Count how many times the letter 'A' appears in the Pinyin representation of the following Chinese sentence: 我们去吃饭
Question payload
{
  "question_text": "Count how many times the letter 'A' appears in the Pinyin representation of the following Chinese sentence: 我们去吃饭",
  "answer_type": "numeric",
  "correct_answer": 1,
  "category": "pinyin_letter_count",
  "difficulty": "medium",
  "tags": [
    "chinese",
    "pinyin",
    "letter_count"
  ],
  "evaluation_criteria": {
    "exact_match": true,
    "case_sensitive": false,
    "contains": false,
    "required_fields": [],
    "tolerance": 0.0,
    "alternatives": []
  }
}

Question
Count how many times the letter 'Q' appears in the Pinyin representation of the following Chinese sentence: 请给我菜单
Question payload
{
  "question_text": "Count how many times the letter 'Q' appears in the Pinyin representation of the following Chinese sentence: 请给我菜单",
  "answer_type": "numeric",
  "correct_answer": 1,
  "category": "pinyin_letter_count",
  "difficulty": "medium",
  "tags": [
    "chinese",
    "pinyin",
    "letter_count"
  ],
  "evaluation_criteria": {
    "exact_match": true,
    "case_sensitive": false,
    "contains": false,
    "required_fields": [],
    "tolerance": 0.0,
    "alternatives": []
  }
}

Question
Count how many times the letter 'S' appears in the Pinyin representation of the following Chinese sentence: 你会说中文吗
Question payload
{
  "question_text": "Count how many times the letter 'S' appears in the Pinyin representation of the following Chinese sentence: 你会说中文吗",
  "answer_type": "numeric",
  "correct_answer": 1,
  "category": "pinyin_letter_count",
  "difficulty": "medium",
  "tags": [
    "chinese",
    "pinyin",
    "letter_count"
  ],
  "evaluation_criteria": {
    "exact_match": true,
    "case_sensitive": false,
    "contains": false,
    "required_fields": [],
    "tolerance": 0.0,
    "alternatives": []
  }
}

Question
Count how many times the letter 'N' appears in the Pinyin representation of the following Chinese sentence: 我住在北京
Question payload
{
  "question_text": "Count how many times the letter 'N' appears in the Pinyin representation of the following Chinese sentence: 我住在北京",
  "answer_type": "numeric",
  "correct_answer": 1,
  "category": "pinyin_letter_count",
  "difficulty": "medium",
  "tags": [
    "chinese",
    "pinyin",
    "letter_count"
  ],
  "evaluation_criteria": {
    "exact_match": true,
    "case_sensitive": false,
    "contains": false,
    "required_fields": [],
    "tolerance": 0.0,
    "alternatives": []
  }
}

Question
Count how many times the letter 'Z' appears in the Pinyin representation of the following Chinese sentence: 这是多少钱
Question payload
{
  "question_text": "Count how many times the letter 'Z' appears in the Pinyin representation of the following Chinese sentence: 这是多少钱",
  "answer_type": "numeric",
  "correct_answer": 1,
  "category": "pinyin_letter_count",
  "difficulty": "medium",
  "tags": [
    "chinese",
    "pinyin",
    "letter_count"
  ],
  "evaluation_criteria": {
    "exact_match": true,
    "case_sensitive": false,
    "contains": false,
    "required_fields": [],
    "tolerance": 0.0,
    "alternatives": []
  }
}

Question
Count how many times the letter 'O' appears in the Pinyin representation of the following Chinese sentence: 帮我开门
Question payload
{
  "question_text": "Count how many times the letter 'O' appears in the Pinyin representation of the following Chinese sentence: 帮我开门",
  "answer_type": "numeric",
  "correct_answer": 1,
  "category": "pinyin_letter_count",
  "difficulty": "medium",
  "tags": [
    "chinese",
    "pinyin",
    "letter_count"
  ],
  "evaluation_criteria": {
    "exact_match": true,
    "case_sensitive": false,
    "contains": false,
    "required_fields": [],
    "tolerance": 0.0,
    "alternatives": []
  }
}

Question
Count how many times the letter 'O' appears in the Pinyin representation of the following Chinese sentence: 今天天气很好
Question payload
{
  "question_text": "Count how many times the letter 'O' appears in the Pinyin representation of the following Chinese sentence: 今天天气很好",
  "answer_type": "numeric",
  "correct_answer": 1,
  "category": "pinyin_letter_count",
  "difficulty": "medium",
  "tags": [
    "chinese",
    "pinyin",
    "letter_count"
  ],
  "evaluation_criteria": {
    "exact_match": true,
    "case_sensitive": false,
    "contains": false,
    "required_fields": [],
    "tolerance": 0.0,
    "alternatives": []
  }
}

Question
Count how many times the letter 'E' appears in the Pinyin representation of the following Chinese sentence: 他的哥哥八岁了
Question payload
{
  "question_text": "Count how many times the letter 'E' appears in the Pinyin representation of the following Chinese sentence: 他的哥哥八岁了",
  "answer_type": "numeric",
  "correct_answer": 4,
  "category": "pinyin_letter_count",
  "difficulty": "medium",
  "tags": [
    "chinese",
    "pinyin",
    "letter_count"
  ],
  "evaluation_criteria": {
    "exact_match": true,
    "case_sensitive": false,
    "contains": false,
    "required_fields": [],
    "tolerance": 0.0,
    "alternatives": []
  }
}

Question
Count how many times the letter 'U' appears in the Pinyin representation of the following Chinese sentence: 我们一起去公园吧
Question payload
{
  "question_text": "Count how many times the letter 'U' appears in the Pinyin representation of the following Chinese sentence: 我们一起去公园吧",
  "answer_type": "numeric",
  "correct_answer": 2,
  "category": "pinyin_letter_count",
  "difficulty": "medium",
  "tags": [
    "chinese",
    "pinyin",
    "letter_count"
  ],
  "evaluation_criteria": {
    "exact_match": true,
    "case_sensitive": false,
    "contains": false,
    "required_fields": [],
    "tolerance": 0.0,
    "alternatives": []
  }
}

Question
Count how many times the letter 'Z' appears in the Pinyin representation of the following Chinese sentence: 这本书非常有趣
Question payload
{
  "question_text": "Count how many times the letter 'Z' appears in the Pinyin representation of the following Chinese sentence: 这本书非常有趣",
  "answer_type": "numeric",
  "correct_answer": 1,
  "category": "pinyin_letter_count",
  "difficulty": "medium",
  "tags": [
    "chinese",
    "pinyin",
    "letter_count"
  ],
  "evaluation_criteria": {
    "exact_match": true,
    "case_sensitive": false,
    "contains": false,
    "required_fields": [],
    "tolerance": 0.0,
    "alternatives": []
  }
}

Question
Count how many times the letter 'T' appears in the Pinyin representation of the following Chinese sentence: 你明天有什么计划
Question payload
{
  "question_text": "Count how many times the letter 'T' appears in the Pinyin representation of the following Chinese sentence: 你明天有什么计划",
  "answer_type": "numeric",
  "correct_answer": 1,
  "category": "pinyin_letter_count",
  "difficulty": "medium",
  "tags": [
    "chinese",
    "pinyin",
    "letter_count"
  ],
  "evaluation_criteria": {
    "exact_match": true,
    "case_sensitive": false,
    "contains": false,
    "required_fields": [],
    "tolerance": 0.0,
    "alternatives": []
  }
}

Question
Count how many times the letter 'X' appears in the Pinyin representation of the following Chinese sentence: 我的猫喜欢睡觉
Question payload
{
  "question_text": "Count how many times the letter 'X' appears in the Pinyin representation of the following Chinese sentence: 我的猫喜欢睡觉",
  "answer_type": "numeric",
  "correct_answer": 1,
  "category": "pinyin_letter_count",
  "difficulty": "medium",
  "tags": [
    "chinese",
    "pinyin",
    "letter_count"
  ],
  "evaluation_criteria": {
    "exact_match": true,
    "case_sensitive": false,
    "contains": false,
    "required_fields": [],
    "tolerance": 0.0,
    "alternatives": []
  }
}

Question
Count how many times the letter 'U' appears in the Pinyin representation of the following Chinese sentence: 中国有很长的历史
Question payload
{
  "question_text": "Count how many times the letter 'U' appears in the Pinyin representation of the following Chinese sentence: 中国有很长的历史",
  "answer_type": "numeric",
  "correct_answer": 2,
  "category": "pinyin_letter_count",
  "difficulty": "medium",
  "tags": [
    "chinese",
    "pinyin",
    "letter_count"
  ],
  "evaluation_criteria": {
    "exact_match": true,
    "case_sensitive": false,
    "contains": false,
    "required_fields": [],
    "tolerance": 0.0,
    "alternatives": []
  }
}

Question
Count how many times the letter 'E' appears in the Pinyin representation of the following Chinese sentence: 昨天我去了图书馆
Question payload
{
  "question_text": "Count how many times the letter 'E' appears in the Pinyin representation of the following Chinese sentence: 昨天我去了图书馆",
  "answer_type": "numeric",
  "correct_answer": 1,
  "category": "pinyin_letter_count",
  "difficulty": "medium",
  "tags": [
    "chinese",
    "pinyin",
    "letter_count"
  ],
  "evaluation_criteria": {
    "exact_match": true,
    "case_sensitive": false,
    "contains": false,
    "required_fields": [],
    "tolerance": 0.0,
    "alternatives": []
  }
}