Syllogism Validity

A benchmark to evaluate whether a model can determine if short categorical syllogisms are logically valid.

Run Benchmark
Questions

16

Leaderboard Entries

23

Best Score

100/100

Leaderboard
Rank Model Score Run Date Actions
Gemma 4 E4B IT (LMStudio)
3200 MB
100
7651.0ms median
 2026-04-02 21:20:48 View Details
2 GPT-5.4 nano
100
974.0ms median
 2026-03-17 19:26:13 View Details
3 GPT-5.4 mini
100
903.5ms median
 2026-03-17 19:04:33 View Details
4 Claude Haiku 4.5
100
2677.0ms median
 2026-03-03 06:01:26 View Details
5 Phi-4 (LMStudio)
9100 MB
100
18457.5ms median
 2026-03-03 01:09:50 View Details
6 GPT-5 nano
100
1534.0ms median
 2026-03-02 18:36:27 View Details
7 GPT-5 mini
100
2448.5ms median
 2026-03-02 18:31:55 View Details
8 Gemma 3 12B (LMStudio)
8100 MB
94
9676.5ms median
 2026-03-02 23:33:34 View Details
9 Gemma 4 12B (LMStudio)
7560 MB
88
10932.5ms median
 2026-06-03 19:34:52 View Details
10 Gemma 2 9B (LMStudio)
5800 MB
88
6058.5ms median
 2026-03-02 23:05:51 View Details
11 Ministral 8B (LMStudio)
4900 MB
75
3686.0ms median
 2026-03-03 00:19:25 View Details
12 Qwen3.5 4B (LMStudio)
3400 MB
69
15473.0ms median
3 latency outliers
2026-03-03 21:37:20 View Details
13 Qwen3.5 2B (LMStudio)
2700 MB
69
8663.0ms median
 2026-03-03 19:27:49 View Details
14 Llama 3.1 8B (LMStudio)
4900 MB
69
5841.0ms median
 2026-03-02 20:38:08 View Details
15 Phi-3.5 Mini (LMStudio)
2500 MB
62
3783.0ms median
 2026-03-03 00:30:04 View Details
16 Llama 3 8B (LMStudio)
4900 MB
62
4346.0ms median
 2026-03-03 00:08:56 View Details
17 OLMo 3 7B (LMStudio)
4300 MB
62
7497.5ms median
 2026-03-02 20:24:48 View Details
18 Granite 3.2 8B (LMStudio)
4900 MB
50
8270.5ms median
 2026-03-02 23:47:52 View Details
19 Llama 3.2 1B (LMStudio)
1300 MB
31
925.5ms median
 2026-03-03 00:14:51 View Details
20 Gemma 2 2B (LMStudio)
1500 MB
19
3209.0ms median
 2026-03-02 22:59:13 View Details
21 SmolLM2 1.7B (LMStudio)
1100 MB
12
1063.0ms median
 2026-03-03 05:51:32 View Details
22 Llama 2 7B (LMStudio)
4900 MB
6
2164.0ms median
 2026-03-03 00:00:16 View Details
23 Gemma 2B (LMStudio)
1500 MB
0
934.5ms median
 2026-03-02 23:12:49 View Details
Questions

Question
Evaluate this syllogism:
Premise 1: All mammals are warm-blooded.
Premise 2: All whales are mammals.
Conclusion: Therefore, all whales are warm-blooded.

Is the conclusion logically valid given the premises?
Question payload
{
  "question_text": "Evaluate this syllogism:\nPremise 1: All mammals are warm-blooded.\nPremise 2: All whales are mammals.\nConclusion: Therefore, all whales are warm-blooded.\n\nIs the conclusion logically valid given the premises?",
  "answer_type": "multiple_choice",
  "correct_answer": "Valid",
  "category": "logic",
  "difficulty": "easy",
  "tags": [
    "knowledge",
    "logic",
    "syllogism"
  ],
  "choices": [
    "Valid",
    "Invalid"
  ],
  "evaluation_criteria": {
    "exact_match": true,
    "case_sensitive": false,
    "contains": false,
    "required_fields": [],
    "tolerance": 0.0,
    "alternatives": []
  }
}

Question
Evaluate this syllogism:
Premise 1: No prime numbers greater than 2 are even.
Premise 2: 11 is a prime number greater than 2.
Conclusion: Therefore, 11 is not even.

Is the conclusion logically valid given the premises?
Question payload
{
  "question_text": "Evaluate this syllogism:\nPremise 1: No prime numbers greater than 2 are even.\nPremise 2: 11 is a prime number greater than 2.\nConclusion: Therefore, 11 is not even.\n\nIs the conclusion logically valid given the premises?",
  "answer_type": "multiple_choice",
  "correct_answer": "Valid",
  "category": "logic",
  "difficulty": "easy",
  "tags": [
    "knowledge",
    "logic",
    "syllogism"
  ],
  "choices": [
    "Valid",
    "Invalid"
  ],
  "evaluation_criteria": {
    "exact_match": true,
    "case_sensitive": false,
    "contains": false,
    "required_fields": [],
    "tolerance": 0.0,
    "alternatives": []
  }
}

Question
Evaluate this syllogism:
Premise 1: Some engineers are musicians.
Premise 2: All musicians are creative.
Conclusion: Therefore, some engineers are creative.

Is the conclusion logically valid given the premises?
Question payload
{
  "question_text": "Evaluate this syllogism:\nPremise 1: Some engineers are musicians.\nPremise 2: All musicians are creative.\nConclusion: Therefore, some engineers are creative.\n\nIs the conclusion logically valid given the premises?",
  "answer_type": "multiple_choice",
  "correct_answer": "Valid",
  "category": "logic",
  "difficulty": "medium",
  "tags": [
    "knowledge",
    "logic",
    "syllogism"
  ],
  "choices": [
    "Valid",
    "Invalid"
  ],
  "evaluation_criteria": {
    "exact_match": true,
    "case_sensitive": false,
    "contains": false,
    "required_fields": [],
    "tolerance": 0.0,
    "alternatives": []
  }
}

Question
Evaluate this syllogism:
Premise 1: All squares are rectangles.
Premise 2: No rectangles are circles.
Conclusion: Therefore, no squares are circles.

Is the conclusion logically valid given the premises?
Question payload
{
  "question_text": "Evaluate this syllogism:\nPremise 1: All squares are rectangles.\nPremise 2: No rectangles are circles.\nConclusion: Therefore, no squares are circles.\n\nIs the conclusion logically valid given the premises?",
  "answer_type": "multiple_choice",
  "correct_answer": "Valid",
  "category": "logic",
  "difficulty": "easy",
  "tags": [
    "knowledge",
    "logic",
    "syllogism"
  ],
  "choices": [
    "Valid",
    "Invalid"
  ],
  "evaluation_criteria": {
    "exact_match": true,
    "case_sensitive": false,
    "contains": false,
    "required_fields": [],
    "tolerance": 0.0,
    "alternatives": []
  }
}

Question
Evaluate this syllogism:
Premise 1: All philosophers are thinkers.
Premise 2: Some thinkers are scientists.
Conclusion: Therefore, some philosophers are scientists.

Is the conclusion logically valid given the premises?
Question payload
{
  "question_text": "Evaluate this syllogism:\nPremise 1: All philosophers are thinkers.\nPremise 2: Some thinkers are scientists.\nConclusion: Therefore, some philosophers are scientists.\n\nIs the conclusion logically valid given the premises?",
  "answer_type": "multiple_choice",
  "correct_answer": "Invalid",
  "category": "logic",
  "difficulty": "hard",
  "tags": [
    "knowledge",
    "logic",
    "syllogism"
  ],
  "choices": [
    "Valid",
    "Invalid"
  ],
  "evaluation_criteria": {
    "exact_match": true,
    "case_sensitive": false,
    "contains": false,
    "required_fields": [],
    "tolerance": 0.0,
    "alternatives": []
  }
}

Question
Evaluate this syllogism:
Premise 1: All planets orbit stars.
Premise 2: Earth is a planet.
Conclusion: Therefore, Earth orbits a star.

Is the conclusion logically valid given the premises?
Question payload
{
  "question_text": "Evaluate this syllogism:\nPremise 1: All planets orbit stars.\nPremise 2: Earth is a planet.\nConclusion: Therefore, Earth orbits a star.\n\nIs the conclusion logically valid given the premises?",
  "answer_type": "multiple_choice",
  "correct_answer": "Valid",
  "category": "logic",
  "difficulty": "easy",
  "tags": [
    "knowledge",
    "logic",
    "syllogism"
  ],
  "choices": [
    "Valid",
    "Invalid"
  ],
  "evaluation_criteria": {
    "exact_match": true,
    "case_sensitive": false,
    "contains": false,
    "required_fields": [],
    "tolerance": 0.0,
    "alternatives": []
  }
}

Question
Evaluate this syllogism:
Premise 1: No insects are mammals.
Premise 2: All bees are insects.
Conclusion: Therefore, no bees are mammals.

Is the conclusion logically valid given the premises?
Question payload
{
  "question_text": "Evaluate this syllogism:\nPremise 1: No insects are mammals.\nPremise 2: All bees are insects.\nConclusion: Therefore, no bees are mammals.\n\nIs the conclusion logically valid given the premises?",
  "answer_type": "multiple_choice",
  "correct_answer": "Valid",
  "category": "logic",
  "difficulty": "easy",
  "tags": [
    "knowledge",
    "logic",
    "syllogism"
  ],
  "choices": [
    "Valid",
    "Invalid"
  ],
  "evaluation_criteria": {
    "exact_match": true,
    "case_sensitive": false,
    "contains": false,
    "required_fields": [],
    "tolerance": 0.0,
    "alternatives": []
  }
}

Question
Evaluate this syllogism:
Premise 1: No reptiles are warm-blooded.
Premise 2: All snakes are reptiles.
Conclusion: Therefore, no snakes are warm-blooded.

Is the conclusion logically valid given the premises?
Question payload
{
  "question_text": "Evaluate this syllogism:\nPremise 1: No reptiles are warm-blooded.\nPremise 2: All snakes are reptiles.\nConclusion: Therefore, no snakes are warm-blooded.\n\nIs the conclusion logically valid given the premises?",
  "answer_type": "multiple_choice",
  "correct_answer": "Valid",
  "category": "logic",
  "difficulty": "easy",
  "tags": [
    "knowledge",
    "logic",
    "syllogism"
  ],
  "choices": [
    "Valid",
    "Invalid"
  ],
  "evaluation_criteria": {
    "exact_match": true,
    "case_sensitive": false,
    "contains": false,
    "required_fields": [],
    "tolerance": 0.0,
    "alternatives": []
  }
}

Question
Evaluate this syllogism:
Premise 1: All doctors have medical degrees.
Premise 2: Sarah has a medical degree.
Conclusion: Therefore, Sarah is a doctor.

Is the conclusion logically valid given the premises?
Question payload
{
  "question_text": "Evaluate this syllogism:\nPremise 1: All doctors have medical degrees.\nPremise 2: Sarah has a medical degree.\nConclusion: Therefore, Sarah is a doctor.\n\nIs the conclusion logically valid given the premises?",
  "answer_type": "multiple_choice",
  "correct_answer": "Invalid",
  "category": "logic",
  "difficulty": "medium",
  "tags": [
    "knowledge",
    "logic",
    "syllogism"
  ],
  "choices": [
    "Valid",
    "Invalid"
  ],
  "evaluation_criteria": {
    "exact_match": true,
    "case_sensitive": false,
    "contains": false,
    "required_fields": [],
    "tolerance": 0.0,
    "alternatives": []
  }
}

Question
Evaluate this syllogism:
Premise 1: Some students are athletes.
Premise 2: All athletes are disciplined.
Conclusion: Therefore, some students are disciplined.

Is the conclusion logically valid given the premises?
Question payload
{
  "question_text": "Evaluate this syllogism:\nPremise 1: Some students are athletes.\nPremise 2: All athletes are disciplined.\nConclusion: Therefore, some students are disciplined.\n\nIs the conclusion logically valid given the premises?",
  "answer_type": "multiple_choice",
  "correct_answer": "Valid",
  "category": "logic",
  "difficulty": "hard",
  "tags": [
    "knowledge",
    "logic",
    "syllogism"
  ],
  "choices": [
    "Valid",
    "Invalid"
  ],
  "evaluation_criteria": {
    "exact_match": true,
    "case_sensitive": false,
    "contains": false,
    "required_fields": [],
    "tolerance": 0.0,
    "alternatives": []
  }
}

Question
Evaluate this syllogism:
Premise 1: All poets are writers.
Premise 2: Some writers are teachers.
Conclusion: Therefore, some poets are teachers.

Is the conclusion logically valid given the premises?
Question payload
{
  "question_text": "Evaluate this syllogism:\nPremise 1: All poets are writers.\nPremise 2: Some writers are teachers.\nConclusion: Therefore, some poets are teachers.\n\nIs the conclusion logically valid given the premises?",
  "answer_type": "multiple_choice",
  "correct_answer": "Invalid",
  "category": "logic",
  "difficulty": "medium",
  "tags": [
    "knowledge",
    "logic",
    "syllogism"
  ],
  "choices": [
    "Valid",
    "Invalid"
  ],
  "evaluation_criteria": {
    "exact_match": true,
    "case_sensitive": false,
    "contains": false,
    "required_fields": [],
    "tolerance": 0.0,
    "alternatives": []
  }
}

Question
Evaluate this syllogism:
Premise 1: Some birds are flightless.
Premise 2: All penguins are birds.
Conclusion: Therefore, some penguins are flightless.

Is the conclusion logically valid given the premises?
Question payload
{
  "question_text": "Evaluate this syllogism:\nPremise 1: Some birds are flightless.\nPremise 2: All penguins are birds.\nConclusion: Therefore, some penguins are flightless.\n\nIs the conclusion logically valid given the premises?",
  "answer_type": "multiple_choice",
  "correct_answer": "Invalid",
  "category": "logic",
  "difficulty": "medium",
  "tags": [
    "knowledge",
    "logic",
    "syllogism"
  ],
  "choices": [
    "Valid",
    "Invalid"
  ],
  "evaluation_criteria": {
    "exact_match": true,
    "case_sensitive": false,
    "contains": false,
    "required_fields": [],
    "tolerance": 0.0,
    "alternatives": []
  }
}

Question
Evaluate this syllogism:
Premise 1: No amphibians are mammals.
Premise 2: All frogs are amphibians.
Conclusion: Therefore, no frogs are mammals.

Is the conclusion logically valid given the premises?
Question payload
{
  "question_text": "Evaluate this syllogism:\nPremise 1: No amphibians are mammals.\nPremise 2: All frogs are amphibians.\nConclusion: Therefore, no frogs are mammals.\n\nIs the conclusion logically valid given the premises?",
  "answer_type": "multiple_choice",
  "correct_answer": "Valid",
  "category": "logic",
  "difficulty": "easy",
  "tags": [
    "knowledge",
    "logic",
    "syllogism"
  ],
  "choices": [
    "Valid",
    "Invalid"
  ],
  "evaluation_criteria": {
    "exact_match": true,
    "case_sensitive": false,
    "contains": false,
    "required_fields": [],
    "tolerance": 0.0,
    "alternatives": []
  }
}

Question
Evaluate this syllogism:
Premise 1: All cats are mammals.
Premise 2: Some pets are cats.
Conclusion: Therefore, some pets are mammals.

Is the conclusion logically valid given the premises?
Question payload
{
  "question_text": "Evaluate this syllogism:\nPremise 1: All cats are mammals.\nPremise 2: Some pets are cats.\nConclusion: Therefore, some pets are mammals.\n\nIs the conclusion logically valid given the premises?",
  "answer_type": "multiple_choice",
  "correct_answer": "Valid",
  "category": "logic",
  "difficulty": "medium",
  "tags": [
    "knowledge",
    "logic",
    "syllogism"
  ],
  "choices": [
    "Valid",
    "Invalid"
  ],
  "evaluation_criteria": {
    "exact_match": true,
    "case_sensitive": false,
    "contains": false,
    "required_fields": [],
    "tolerance": 0.0,
    "alternatives": []
  }
}

Question
Evaluate this syllogism:
Premise 1: All novels are books.
Premise 2: Some books are illustrated.
Conclusion: Therefore, some novels are illustrated.

Is the conclusion logically valid given the premises?
Question payload
{
  "question_text": "Evaluate this syllogism:\nPremise 1: All novels are books.\nPremise 2: Some books are illustrated.\nConclusion: Therefore, some novels are illustrated.\n\nIs the conclusion logically valid given the premises?",
  "answer_type": "multiple_choice",
  "correct_answer": "Invalid",
  "category": "logic",
  "difficulty": "medium",
  "tags": [
    "knowledge",
    "logic",
    "syllogism"
  ],
  "choices": [
    "Valid",
    "Invalid"
  ],
  "evaluation_criteria": {
    "exact_match": true,
    "case_sensitive": false,
    "contains": false,
    "required_fields": [],
    "tolerance": 0.0,
    "alternatives": []
  }
}

Question
Evaluate this syllogism:
Premise 1: All painters are artists.
Premise 2: Some artists are musicians.
Conclusion: Therefore, some painters are musicians.

Is the conclusion logically valid given the premises?
Question payload
{
  "question_text": "Evaluate this syllogism:\nPremise 1: All painters are artists.\nPremise 2: Some artists are musicians.\nConclusion: Therefore, some painters are musicians.\n\nIs the conclusion logically valid given the premises?",
  "answer_type": "multiple_choice",
  "correct_answer": "Invalid",
  "category": "logic",
  "difficulty": "medium",
  "tags": [
    "knowledge",
    "logic",
    "syllogism"
  ],
  "choices": [
    "Valid",
    "Invalid"
  ],
  "evaluation_criteria": {
    "exact_match": true,
    "case_sensitive": false,
    "contains": false,
    "required_fields": [],
    "tolerance": 0.0,
    "alternatives": []
  }
}