How to test whether an AI model really reasons: the contradiction-trap eval

MModel releases now routinely promise better "reasoning." The word has become a marketing term, and most public benchmarks cannot tell you whether a model reasons or has simply seen enough similar problems to produce a convincing answer. If you depend on a model for analysis, planning or any task where a wrong answer is expensive, you need a test that punishes pattern matching.

The contradiction-trap eval is one of the simplest tests that does this. You give a model a problem that looks solvable but contains a planted logical contradiction, and you score one thing: does the model notice that the problem cannot be solved as stated? This guide explains why the design works, how to build your own version in a day, how to score it fairly, and what the common failure patterns tell you about a model.

Why ordinary benchmarks miss reasoning failures

Most reasoning benchmarks reward the final answer. A math word problem has one correct number; a multiple-choice question has one correct letter. That design has two blind spots.

• Contamination. Popular benchmark questions, or close paraphrases, leak into training data. A model can score well because it has memorised the shape of the solution. See our guide on checking benchmark contamination for how widespread this is.
• Solvability is assumed. Every benchmark item has an answer, so the model is never rewarded for saying "this does not work." A system that always produces a fluent solution looks perfect, right up until you give it a broken input in production.

Real work is full of broken inputs: requirements that conflict, data that contradicts itself, instructions written by two people who never talked. A model that cannot detect an inconsistency will confidently resolve it in an arbitrary direction, and that is the most expensive kind of error because nobody sees it.

What a contradiction trap is

A contradiction trap is a long, self-contained logic puzzle (scheduling, seating, resource allocation or rule-following) with three properties:

1. It looks routine. The format resembles thousands of puzzles the model has seen, which invites pattern matching.
2. It contains exactly one planted contradiction. Two rules, usually far apart in the prompt, cannot both be true. For example, rule 4 says "the audit happens before the migration" and rule 17 says "the migration finishes before any review step, and the audit is a review step."
3. The only correct response is to flag the conflict. Any "solution" is wrong by construction, because it must silently drop one of the rules.

Because the puzzle is new, memorisation cannot help. Because it is long, the model has to track state across many tokens. And because the correct answer is a refusal to solve, fluent confidence earns nothing.

How to build your own trap in an afternoon

You do not need a research team. You need a domain your users care about and a few hours.

Step 1: write a solvable base puzzle

Start with a puzzle that has a unique, verifiable solution. Fifteen to twenty-five rules is a good size. Verify the solution with a constraint solver or a short script, not by eye; it is easy to write a puzzle that accidentally has zero or many solutions.

from itertools import permutations

TASKS = ["audit", "backup", "migration", "review", "deploy"]

def valid(order):
    pos = {t: i for i, t in enumerate(order)}
    rules = [
        pos["backup"] < pos["migration"],
        pos["audit"] < pos["deploy"],
        pos["review"] == pos["deploy"] - 1,
        # ...the rest of your rules
    ]
    return all(rules)

solutions = [o for o in permutations(TASKS) if valid(o)]
print(len(solutions), "solution(s)")   # must print exactly 1 for the base puzzle

Step 2: plant the contradiction

Add one rule that conflicts with an existing rule, and place it far away from the rule it contradicts, ideally in the last third of the prompt. Re-run the solver: it should now report zero solutions. If it still finds one, your "contradiction" is not one.

Step 3: pad with realistic context

Wrap the rules in the kind of material a real task would include: background, a few irrelevant constraints, formatting instructions. Padding tests whether the model can find the conflict among noise, which is closer to how contradictions show up in real documents. Long prompts also expose the position effects covered in why long context forgets the middle.

Step 4: make variants

One trap is an anecdote. Build at least 20 variants by changing the domain, the position of the planted rule and the distance between the conflicting rules. Keep a few solvable control puzzles in the set too, so that a model which flags *everything* as contradictory does not get a perfect score.

How to score it without fooling yourself

Scoring is where most homemade evals go wrong. Three rules keep it honest.

Score the final answer blind. Strip model names and formatting before anyone grades. Humans are biased towards outputs that "look smart," and they recognise a model's house style quickly.

Use a strict rubric with no partial credit. For trap puzzles, a response passes only if it identifies the two conflicting rules and declines to present a solution as valid. For control puzzles, it passes only with the correct solution.

Run each item several times. Sampling makes outputs vary even at low temperature. Three to five runs per item show whether a model catches the trap consistently or only sometimes, and "sometimes" is not a property you can ship. Before you compare two models, read why a 2-point improvement might be noise and compute an interval.

for model in MODELS:
    for run in range(3):
        out = client.complete(model, PROMPT, temperature=0.2)
        scores[model].append(judge(out, rubric="strict"))

If you use a model as one of the judges, calibrate it against human labels first; an uncalibrated judge often rewards the confident wrong solution. Our guide to calibrating model graders covers the process.

The four failure patterns you will see

When you read the transcripts, and you should read them rather than only the score, failures tend to fall into recognisable shapes.

1. Confident fabrication. The model presents a full solution and never mentions a conflict. Usually it has quietly violated one rule.
2. Silent rule-dropping. The model's reasoning restates the rules but omits the planted one, as if it was never there. This is common when the conflict sits in the middle of a long prompt.
3. Notice-and-proceed. The most revealing failure: the model writes something like "Note: rules 4 and 17 appear to conflict. Proceeding with the solution." It detected the problem and then prioritised producing an answer anyway.
4. Over-flagging. The model reports contradictions in solvable control puzzles. This often appears after you add a "check for conflicts" instruction, so always keep controls in the set.

Notice-and-proceed tells you something important: the model has the *capability* to detect the issue, but its training rewards finishing the task. That is a prompting problem you can often fix.

A prompt change that usually helps

Because many failures are notice-and-proceed, an explicit stopping rule tends to improve detection. Add it to the system prompt and re-run the whole set, controls included.

Measure the false-alarm rate on control puzzles alongside detection. A prompt that raises detection while doubling false alarms has moved the problem, not solved it. Self-checking wrappers like the red-team-yourself prompt work on the same principle and have the same trade-off.

How to use the results

A contradiction-trap score is not a general intelligence rating. It measures one specific, valuable behaviour: noticing when a task is ill-posed. Use it to:

• choose between candidate models for analysis, planning, legal or compliance work, where silent inconsistency is costly;
• decide whether you need a validation step, a second model or a human review before acting on outputs;
• regression-test prompt and model upgrades, because detection behaviour can change between versions without any announcement.

Keep the trap set private. Once a puzzle is published it stops being new, and it slowly turns into one more contaminated benchmark.

Frequently asked questions

What is a contradiction-trap eval?

It is a test in which a model receives a puzzle or task that contains a deliberately planted logical contradiction. The only correct response is to identify the conflicting rules and decline to solve. It measures whether a model tracks constraints rather than pattern-matching to a familiar solution.

How many test items do I need?

Aim for at least 20 trap variants plus 10 or more solvable control puzzles, each run three to five times. Fewer items make the score too noisy to compare models, and without controls you cannot detect a model that flags everything.

Does a higher temperature make models miss contradictions?

Higher temperatures increase run-to-run variation, so detection becomes less consistent. Run your eval at the temperature you use in production, and report consistency across runs, not just the average.

Can I publish my trap puzzles?

You can, but published items gradually enter training data and lose their value as a test. Keep a private held-out set for decisions and publish only examples that illustrate the method.