Same question 10 times, different answers. Which one passes? AI evals and guardrails

AI evals and guardrails can share checks: evals measure performance, while guardrails apply checks before delivery. In our seeded simulation a retry-once guardrail with three code checks still delivered 77 wrong replies out of 930 (8.3%), each with the right $40 and a reason the policy never gives, because no check compared the reason with the policy.

How do AI evals and guardrails use the same checks?

An eval runs the check function over many saved replies and reports a score. A guardrail calls the same function on one live reply and acts on the result.

Why does the same question get different answers?

With sampling enabled, decoding selects tokens from a probability distribution. Greedy decoding chooses a highest-probability token. Lower temperature concentrates sampling probabilities; repeatability also depends on the serving implementation. The Hugging Face generation guide describes greedy search and multinomial sampling side by side. With sampling, repeated asks can return the same fact in different words, keys in a different order, a code fence around the JSON, a sentence before it, and now and then a wrong number. A system that acts on the reply (issue a refund, call a tool) needs a rule for which replies count.

That rule is a contract. Here it is "return {decision, amount, reason} as bare JSON". The JSON Schema specification is the standard way to write such a contract down. Once it is written, you can test every reply against it.

What our lab tested

We did not call a model. We wrote 13 kinds of reply by hand, gave each a fixed weight, and drew from them with a seeded random number, so the run is repeatable. The question is a refund: an $80 item plus $8 shipping, bought 20 days ago. The policy pays 50% of the item from day 15 to day 30 and never refunds shipping, so the right answer is partial, $40.

The eval has three code checks:

format   valid JSON, decision in {full, partial, none}, amount is a number, reason not empty
math     decision and amount equal the policy, in whole cents
promise  reject reasons matching the configured forbidden-phrase regex; paraphrases can pass

The guardrail calls the same function. A failing reply is blocked and the question is asked once more; if the retry fails too, the answer goes to a person. Each reply also carries a hidden label we set by hand, "correct", which the checks never see. That label is what we measure the checks against.

The code is labs/ai-17/run.mjs and every attempt with its check results is in labs/ai-17/replies.json. Run node labs/ai-17/run.mjs (Node 22, no packages, no network). The numbers below come from the saved run of 2026-11-06.

Guardrail or no guardrail: the race

We ran the same 10 asks down two lanes on the lab's simulated clock (each model call takes 700 to 1,300 ms on a virtual clock, nothing waits).

Lane	Model calls	Simulated time	Correct answers delivered
A · no check	10	9,910 ms	7 of 10
B · guardrail, retry once	13	12,593 ms	10 of 10

first tries:      7 of 10 pass, 3 fail (format: 2 fenced JSON, 1 missing reason)
retries:          3, all pass
extra time:       12,593 - 9,910 = 2,683 ms over 10 asks = 268.3 ms per ask
worst single ask: 1,065 ms extra (one full model call)
check time:       about 0.02 ms per reply, measured; tiny next to a model call

In this simulation, retries dominate the added latency. Our retry independently resamples the same authored distribution, with no feedback about what failed. With first-attempt rejection probability r, one retry uses 1 + r calls per request on average; independent attempts leave r² requests blocked. The long run fits: 73.6% passed first time, so r = 0.264, giving 1.264 calls per request (1,264 attempts for 1,000 asks) and 0.264² ≈ 7.0% blocked (70 of 1,000). Real failures may be correlated: a prompt that confuses the model once tends to confuse it again, so the true blocked rate can sit well above r².

This is the classic retry pattern with a check deciding when (Part 80 of the series covers retries in depth): retry only what failed, cap the attempts, and send the rest somewhere safe. In production you add a budget, as the AWS Builders' Library explains, so retries cannot multiply load during an outage.

Two of our three "failures" were fenced JSON whose reasoning was fine. Many real apps strip code fences before parsing, which would turn those into passes; our format check is strict on purpose, to show the mechanism.

Which wrong reply gets through?

Over 1,000 draws (seed 117) there were 1,264 attempts, 411 of them wrong by the hidden label. The guardrail rejected 334 incorrect attempts and accepted 77.

accepted wrong attempts = 77 / 411 = 18.7%   (retries included)
delivered               = 853 correct + 77 incorrect = 930; 70 blocked after retry
incorrect among delivered = 77 / 930 = 8.3%
every one of the 77 is this reply type:
{"decision":"partial","amount":40,"reason":"You are a loyal customer, so we can refund half."}

The 18.7% figure measures accepted attempts among incorrect attempts. The 8.3% is what a customer sees. The accepted reply has valid JSON, the right decision, exactly 4,000 cents and no phrase from the forbidden list, so all three checks pass. The reason is wrong. The policy pays half because of the purchase day, and loyalty plays no part in it. A customer who reads it learns a rule that does not exist and may expect it next time.

The same blind spot shows up whenever one AI writes and another reviews: the reviewer only catches what its own test looks at. It is also why a quote an AI cites can be missing from its source while the citation format looks perfect.

How to close the gap

Compute the decision and a reason code in trusted policy code, then render an approved explanation. For this refund, the code knows the order is on day 20, so it emits PARTIAL_DAY_15_30 and the reply uses the matching approved sentence. The model never writes the reason, so it cannot invent one.

For free-form explanations, calibrate a model grader against human-labelled examples. Give it the policy text and a narrow yes or no question ("does this reason follow from the policy?"), then measure its agreement with people on a labelled set before you trust it. It costs another call and has its own error rate.

After delivery, pull a random sample of real replies every day and have a person label them. This finds the holes nobody thought to write a check for. Anthropic's guide to evals for agents describes code-based outcome checks, model graders that need calibration with human graders, and human review. When a sample reveals a new kind of failure, turn it into a check, add it to the eval set, and rerun.

Trade-offs

Option	Good for	Costs you	Use when
No check	Lowest latency	Every bad reply reaches the user	Drafts a person always edits
Code checks + retry once	Format, math, listed phrases	One extra call per failure	Any reply a program acts on
Reason code from policy code	Explanations that must match the rule exactly	Writing and approving each sentence	Decisions with a fixed policy
Calibrated model grader	Free-form explanations	An extra call on every reply; its own error rate	Replies too varied for fixed text
Human review of samples	Finding failure kinds you did not foresee	People's time, found after delivery	Always, at a small daily rate

What we saw when we ran it

We ran node labs/ai-17/run.mjs on a Mac on 2026-11-06. The output begins with a simulation notice: authored replies, and no model is called. Then come 10 asks of the same refund question, each with its reply category and a PASS or FAIL. Asks #1, #5 and #8 failed the format check, retried, and came back correct, so the short run ended at 7/10 passing first try and 10/10 delivered correct.

Our terminal after the run: the 10 asks, the retries and the 1000-ask summary.

We read the long run twice. Over 1000 asks the guardrail blocked 334 of 411 incorrect attempts, accepted 77, and blocked 0 correct ones. Those 77 replies had a valid format, the correct amount and no forbidden phrase, but wrong reasoning, and since the screen notes that guardrail and eval use the same checks, the eval passed them too. A green eval vouches only for what its checks read.

Anthropic's research page on evals for AI agents.

That page describes code-based outcome checks, model graders that need calibration with human graders, and human review, the mix we point to in the section above.

Our take

Run your checks before delivery as well as in a report. In our run that turned three first-try failures into three retries, and every delivered answer in the 10-ask run was correct. The green ticks still hid 77 wrong replies in the long run, and we only saw them because of a hidden label; in a real system that label comes from people reading samples. If a reply explains a decision, let policy code pick the reason. The numbers here come from replies we wrote, not from any real model, so use them to understand the effect and measure your own rates. Related reads: why an AI obeyed a hidden line and whether an agent would delete your files.

This article describes general patterns and our own simulation. It is not a description of any company's private system, and no model was called in the lab.