Pinyin Letter Count

A benchmark to evaluate a model's ability to count how many times a specific letter appears in the Pinyin representation of a Chinese sentence.

Run Benchmark
Questions

20

Total Runs

19

Best Score

60/100

Leaderboard
Rank Model Score Run Date Actions
Gemma 2B (LMStudio)
1500 MB
60
2026-02-28 15:34:16 View Details
2 Gemma 3 12B (LMStudio)
8100 MB
50
2026-02-28 15:50:17 View Details
3 GPT-5 mini
35
2026-02-26 01:23:39 View Details
4 SmolLM2 1.7B (LMStudio)
1100 MB
35
2026-02-28 19:18:46 View Details
5 Claude Haiku 4.5
35
2026-02-28 19:43:34 View Details
6 Gemma 2 2B (LMStudio)
1500 MB
30
2026-02-28 03:21:28 View Details
7 Gemma 2 9B (LMStudio)
5800 MB
30
2026-02-28 14:59:17 View Details
8 Granite 3.2 8B (LMStudio)
4900 MB
25
2026-02-28 16:17:13 View Details
9 Llama 3 8B (LMStudio)
4900 MB
25
2026-02-28 16:45:15 View Details
10 OLMo 3 7B (LMStudio)
4300 MB
20
2026-02-28 17:36:07 View Details
11 Qwen3 VL 8B (LMStudio)
5000 MB
20
2026-02-28 18:35:24 View Details
12 GPT-5 nano
15
2026-02-26 00:56:32 View Details
13 Llama 2 7B (LMStudio)
4900 MB
15
2026-02-28 03:04:01 View Details
14 Llama 3.1 8B (LMStudio)
4900 MB
15
2026-02-28 17:00:38 View Details
15 Ministral 8B (LMStudio)
4900 MB
15
2026-02-28 17:21:45 View Details
16 Llama 3.2 1B (LMStudio)
1300 MB
10
2026-02-28 17:12:30 View Details
17 Qwen3 1.7B (LMStudio)
1100 MB
5
2026-02-28 18:04:43 View Details
18 Qwen3 4B (LMStudio)
2800 MB
5
2026-02-28 18:12:27 View Details
19 Phi-3.5 Mini (LMStudio)
2500 MB
0
2026-02-28 17:48:31 View Details
Questions

Question
Count how many times the letter 'U' appears in the Pinyin representation of the following Chinese sentence: 我喜欢学习中文
Question payload
{
  "question_text": "Count how many times the letter 'U' appears in the Pinyin representation of the following Chinese sentence: 我喜欢学习中文",
  "answer_type": "numeric",
  "correct_answer": 2,
  "category": "pinyin_letter_count",
  "difficulty": "medium",
  "tags": [
    "chinese",
    "pinyin",
    "letter_count"
  ],
  "evaluation_criteria": {
    "exact_match": true,
    "case_sensitive": false,
    "contains": false,
    "required_fields": [],
    "tolerance": 0.0,
    "alternatives": []
  }
}

Question
Count how many times the letter 'E' appears in the Pinyin representation of the following Chinese sentence: 北京是中国的首都
Question payload
{
  "question_text": "Count how many times the letter 'E' appears in the Pinyin representation of the following Chinese sentence: 北京是中国的首都",
  "answer_type": "numeric",
  "correct_answer": 2,
  "category": "pinyin_letter_count",
  "difficulty": "medium",
  "tags": [
    "chinese",
    "pinyin",
    "letter_count"
  ],
  "evaluation_criteria": {
    "exact_match": true,
    "case_sensitive": false,
    "contains": false,
    "required_fields": [],
    "tolerance": 0.0,
    "alternatives": []
  }
}

Question
Count how many times the letter 'J' appears in the Pinyin representation of the following Chinese sentence: 你叫什么名字
Question payload
{
  "question_text": "Count how many times the letter 'J' appears in the Pinyin representation of the following Chinese sentence: 你叫什么名字",
  "answer_type": "numeric",
  "correct_answer": 1,
  "category": "pinyin_letter_count",
  "difficulty": "medium",
  "tags": [
    "chinese",
    "pinyin",
    "letter_count"
  ],
  "evaluation_criteria": {
    "exact_match": true,
    "case_sensitive": false,
    "contains": false,
    "required_fields": [],
    "tolerance": 0.0,
    "alternatives": []
  }
}

Question
Count how many times the letter 'U' appears in the Pinyin representation of the following Chinese sentence: 我想喝水
Question payload
{
  "question_text": "Count how many times the letter 'U' appears in the Pinyin representation of the following Chinese sentence: 我想喝水",
  "answer_type": "numeric",
  "correct_answer": 1,
  "category": "pinyin_letter_count",
  "difficulty": "medium",
  "tags": [
    "chinese",
    "pinyin",
    "letter_count"
  ],
  "evaluation_criteria": {
    "exact_match": true,
    "case_sensitive": false,
    "contains": false,
    "required_fields": [],
    "tolerance": 0.0,
    "alternatives": []
  }
}

Question
Count how many times the letter 'H' appears in the Pinyin representation of the following Chinese sentence: 他在看书
Question payload
{
  "question_text": "Count how many times the letter 'H' appears in the Pinyin representation of the following Chinese sentence: 他在看书",
  "answer_type": "numeric",
  "correct_answer": 1,
  "category": "pinyin_letter_count",
  "difficulty": "medium",
  "tags": [
    "chinese",
    "pinyin",
    "letter_count"
  ],
  "evaluation_criteria": {
    "exact_match": true,
    "case_sensitive": false,
    "contains": false,
    "required_fields": [],
    "tolerance": 0.0,
    "alternatives": []
  }
}

Question
Count how many times the letter 'T' appears in the Pinyin representation of the following Chinese sentence: 今天很热
Question payload
{
  "question_text": "Count how many times the letter 'T' appears in the Pinyin representation of the following Chinese sentence: 今天很热",
  "answer_type": "numeric",
  "correct_answer": 1,
  "category": "pinyin_letter_count",
  "difficulty": "medium",
  "tags": [
    "chinese",
    "pinyin",
    "letter_count"
  ],
  "evaluation_criteria": {
    "exact_match": true,
    "case_sensitive": false,
    "contains": false,
    "required_fields": [],
    "tolerance": 0.0,
    "alternatives": []
  }
}

Question
Count how many times the letter 'A' appears in the Pinyin representation of the following Chinese sentence: 我们去吃饭
Question payload
{
  "question_text": "Count how many times the letter 'A' appears in the Pinyin representation of the following Chinese sentence: 我们去吃饭",
  "answer_type": "numeric",
  "correct_answer": 1,
  "category": "pinyin_letter_count",
  "difficulty": "medium",
  "tags": [
    "chinese",
    "pinyin",
    "letter_count"
  ],
  "evaluation_criteria": {
    "exact_match": true,
    "case_sensitive": false,
    "contains": false,
    "required_fields": [],
    "tolerance": 0.0,
    "alternatives": []
  }
}

Question
Count how many times the letter 'Q' appears in the Pinyin representation of the following Chinese sentence: 请给我菜单
Question payload
{
  "question_text": "Count how many times the letter 'Q' appears in the Pinyin representation of the following Chinese sentence: 请给我菜单",
  "answer_type": "numeric",
  "correct_answer": 1,
  "category": "pinyin_letter_count",
  "difficulty": "medium",
  "tags": [
    "chinese",
    "pinyin",
    "letter_count"
  ],
  "evaluation_criteria": {
    "exact_match": true,
    "case_sensitive": false,
    "contains": false,
    "required_fields": [],
    "tolerance": 0.0,
    "alternatives": []
  }
}

Question
Count how many times the letter 'S' appears in the Pinyin representation of the following Chinese sentence: 你会说中文吗
Question payload
{
  "question_text": "Count how many times the letter 'S' appears in the Pinyin representation of the following Chinese sentence: 你会说中文吗",
  "answer_type": "numeric",
  "correct_answer": 1,
  "category": "pinyin_letter_count",
  "difficulty": "medium",
  "tags": [
    "chinese",
    "pinyin",
    "letter_count"
  ],
  "evaluation_criteria": {
    "exact_match": true,
    "case_sensitive": false,
    "contains": false,
    "required_fields": [],
    "tolerance": 0.0,
    "alternatives": []
  }
}

Question
Count how many times the letter 'N' appears in the Pinyin representation of the following Chinese sentence: 我住在北京
Question payload
{
  "question_text": "Count how many times the letter 'N' appears in the Pinyin representation of the following Chinese sentence: 我住在北京",
  "answer_type": "numeric",
  "correct_answer": 1,
  "category": "pinyin_letter_count",
  "difficulty": "medium",
  "tags": [
    "chinese",
    "pinyin",
    "letter_count"
  ],
  "evaluation_criteria": {
    "exact_match": true,
    "case_sensitive": false,
    "contains": false,
    "required_fields": [],
    "tolerance": 0.0,
    "alternatives": []
  }
}

Question
Count how many times the letter 'Z' appears in the Pinyin representation of the following Chinese sentence: 这是多少钱
Question payload
{
  "question_text": "Count how many times the letter 'Z' appears in the Pinyin representation of the following Chinese sentence: 这是多少钱",
  "answer_type": "numeric",
  "correct_answer": 1,
  "category": "pinyin_letter_count",
  "difficulty": "medium",
  "tags": [
    "chinese",
    "pinyin",
    "letter_count"
  ],
  "evaluation_criteria": {
    "exact_match": true,
    "case_sensitive": false,
    "contains": false,
    "required_fields": [],
    "tolerance": 0.0,
    "alternatives": []
  }
}

Question
Count how many times the letter 'O' appears in the Pinyin representation of the following Chinese sentence: 帮我开门
Question payload
{
  "question_text": "Count how many times the letter 'O' appears in the Pinyin representation of the following Chinese sentence: 帮我开门",
  "answer_type": "numeric",
  "correct_answer": 1,
  "category": "pinyin_letter_count",
  "difficulty": "medium",
  "tags": [
    "chinese",
    "pinyin",
    "letter_count"
  ],
  "evaluation_criteria": {
    "exact_match": true,
    "case_sensitive": false,
    "contains": false,
    "required_fields": [],
    "tolerance": 0.0,
    "alternatives": []
  }
}

Question
Count how many times the letter 'O' appears in the Pinyin representation of the following Chinese sentence: 今天天气很好
Question payload
{
  "question_text": "Count how many times the letter 'O' appears in the Pinyin representation of the following Chinese sentence: 今天天气很好",
  "answer_type": "numeric",
  "correct_answer": 1,
  "category": "pinyin_letter_count",
  "difficulty": "medium",
  "tags": [
    "chinese",
    "pinyin",
    "letter_count"
  ],
  "evaluation_criteria": {
    "exact_match": true,
    "case_sensitive": false,
    "contains": false,
    "required_fields": [],
    "tolerance": 0.0,
    "alternatives": []
  }
}

Question
Count how many times the letter 'E' appears in the Pinyin representation of the following Chinese sentence: 他的哥哥八岁了
Question payload
{
  "question_text": "Count how many times the letter 'E' appears in the Pinyin representation of the following Chinese sentence: 他的哥哥八岁了",
  "answer_type": "numeric",
  "correct_answer": 4,
  "category": "pinyin_letter_count",
  "difficulty": "medium",
  "tags": [
    "chinese",
    "pinyin",
    "letter_count"
  ],
  "evaluation_criteria": {
    "exact_match": true,
    "case_sensitive": false,
    "contains": false,
    "required_fields": [],
    "tolerance": 0.0,
    "alternatives": []
  }
}

Question
Count how many times the letter 'U' appears in the Pinyin representation of the following Chinese sentence: 我们一起去公园吧
Question payload
{
  "question_text": "Count how many times the letter 'U' appears in the Pinyin representation of the following Chinese sentence: 我们一起去公园吧",
  "answer_type": "numeric",
  "correct_answer": 2,
  "category": "pinyin_letter_count",
  "difficulty": "medium",
  "tags": [
    "chinese",
    "pinyin",
    "letter_count"
  ],
  "evaluation_criteria": {
    "exact_match": true,
    "case_sensitive": false,
    "contains": false,
    "required_fields": [],
    "tolerance": 0.0,
    "alternatives": []
  }
}

Question
Count how many times the letter 'Z' appears in the Pinyin representation of the following Chinese sentence: 这本书非常有趣
Question payload
{
  "question_text": "Count how many times the letter 'Z' appears in the Pinyin representation of the following Chinese sentence: 这本书非常有趣",
  "answer_type": "numeric",
  "correct_answer": 1,
  "category": "pinyin_letter_count",
  "difficulty": "medium",
  "tags": [
    "chinese",
    "pinyin",
    "letter_count"
  ],
  "evaluation_criteria": {
    "exact_match": true,
    "case_sensitive": false,
    "contains": false,
    "required_fields": [],
    "tolerance": 0.0,
    "alternatives": []
  }
}

Question
Count how many times the letter 'T' appears in the Pinyin representation of the following Chinese sentence: 你明天有什么计划
Question payload
{
  "question_text": "Count how many times the letter 'T' appears in the Pinyin representation of the following Chinese sentence: 你明天有什么计划",
  "answer_type": "numeric",
  "correct_answer": 1,
  "category": "pinyin_letter_count",
  "difficulty": "medium",
  "tags": [
    "chinese",
    "pinyin",
    "letter_count"
  ],
  "evaluation_criteria": {
    "exact_match": true,
    "case_sensitive": false,
    "contains": false,
    "required_fields": [],
    "tolerance": 0.0,
    "alternatives": []
  }
}

Question
Count how many times the letter 'X' appears in the Pinyin representation of the following Chinese sentence: 我的猫喜欢睡觉
Question payload
{
  "question_text": "Count how many times the letter 'X' appears in the Pinyin representation of the following Chinese sentence: 我的猫喜欢睡觉",
  "answer_type": "numeric",
  "correct_answer": 1,
  "category": "pinyin_letter_count",
  "difficulty": "medium",
  "tags": [
    "chinese",
    "pinyin",
    "letter_count"
  ],
  "evaluation_criteria": {
    "exact_match": true,
    "case_sensitive": false,
    "contains": false,
    "required_fields": [],
    "tolerance": 0.0,
    "alternatives": []
  }
}

Question
Count how many times the letter 'U' appears in the Pinyin representation of the following Chinese sentence: 中国有很长的历史
Question payload
{
  "question_text": "Count how many times the letter 'U' appears in the Pinyin representation of the following Chinese sentence: 中国有很长的历史",
  "answer_type": "numeric",
  "correct_answer": 2,
  "category": "pinyin_letter_count",
  "difficulty": "medium",
  "tags": [
    "chinese",
    "pinyin",
    "letter_count"
  ],
  "evaluation_criteria": {
    "exact_match": true,
    "case_sensitive": false,
    "contains": false,
    "required_fields": [],
    "tolerance": 0.0,
    "alternatives": []
  }
}

Question
Count how many times the letter 'E' appears in the Pinyin representation of the following Chinese sentence: 昨天我去了图书馆
Question payload
{
  "question_text": "Count how many times the letter 'E' appears in the Pinyin representation of the following Chinese sentence: 昨天我去了图书馆",
  "answer_type": "numeric",
  "correct_answer": 1,
  "category": "pinyin_letter_count",
  "difficulty": "medium",
  "tags": [
    "chinese",
    "pinyin",
    "letter_count"
  ],
  "evaluation_criteria": {
    "exact_match": true,
    "case_sensitive": false,
    "contains": false,
    "required_fields": [],
    "tolerance": 0.0,
    "alternatives": []
  }
}