Why Does ChatGPT Fail These Questions?

AI language models like ChatGPT (GPT-4o) are trained to predict the next token in a sequence — not to reason from first principles. This makes them surprisingly bad at tasks that seem simple to humans:

• Counting letters — AI processes text as "tokens" (chunks), not individual characters. "Strawberry" becomes "straw" + "berry", hiding its R's.
• CRT math traps — The Cognitive Reflection Test is designed to trigger an intuitive wrong answer. AI models trained on text inherit these same intuitions from human-written data.
• Logic riddles with hidden assumptions — If the setup of a riddle requires catching a subtle implication ("he's dead"), AI often misses it and answers the surface-level question.
• ARC-AGI patterns — ARC-AGI tests the ability to learn new rules from a few examples. Base frontier models score near 0% because they can't generalize the way humans can.

What Is ARC-AGI?

ARC-AGI (Abstraction and Reasoning Corpus for Artificial General Intelligence) was designed by deep learning pioneer François Chollet as a benchmark for general intelligence — not just memorized knowledge. Each puzzle shows a few input→output grid examples, and you must infer the rule to complete a new test case.

When ARC-AGI-2 launched in March 2025, every frontier AI model — GPT-4o, Claude 3.7, Gemini 2.0 Flash — scored between 0% and 1.3%. The human average is 60%. Only with extremely expensive multi-attempt scaffolding (costing $30–$77 per question) did AI systems approach human performance. The ARC-AGI dataset is released under the Apache 2.0 license by the ARC Prize Foundation.

How Do You Score vs. ChatGPT?

GPT-4o (without extended thinking) scores approximately 1/10 on the questions in this challenge, based on documented research and widely reported failure cases from 2024–2025. The one question it reliably gets right (the 28-day riddle) is now well-known enough that it appears in GPT's training data. On novel variants of all the other questions, failure rates range from 40–100%.

Most humans who approach these questions carefully score 6–8/10. The ARC puzzle and the "portrait" logic riddle are the hardest — they trip up humans and AI alike, though for different reasons.