Research7 min read
One AI judge found 14 violations. A second found 8. They agreed on 1.
The Draimo team
Short answer
If you evaluate AI with one AI judge, you publish that judge's opinion. In the October 2026 run of Overlooked Bench, an LLM judge flagged 14 broken constraints and an independent verifier flagged 8. They agreed on one, a Cohen's kappa of 0.06. A second judge from another provider then reordered the leaderboard. Use two instruments, publish where they disagree, and count only what both find.
Try it in Draimo
Design an evaluation you can defend
- Design an evaluation for our customer support assistant: two judges from different providers, a rubric with four dimensions, ten calibration questions with known answers, and how we report where the judges disagree.
- Turn this policy into yes or no checks an AI answer can be scored against, and say which ones a second reviewer should confirm.
- Documents
- Turns the result into a Word, PDF or PowerPoint file your team can review and sign off.
- Draimo Pro
- Agent Mode
- Researches current public information and cites every source, so each claim in a plan or review can be checked.
- Monthly and yearly plans
- Files
- Attach the design doc, runbook or policy and ask Draimo to check it line by line.
- One file per message on every plan; several at once on paid plans
What happened
Overlooked Bench's production engineering track checks every answer against its stated constraints: 315 yes or no checks across seven models. Each check was answered twice, once by the LLM judge, DeepSeek V3.1, and once by an independent verifier model built for exactly this kind of decision.
The judge, on its own, said 14 constraints were broken. The verifier said 8. They agreed on one. A benchmark run with only the judge would have published 14; one run with only the verifier would have published 8. The run published 1, the only violation both found, with the offending sentence quoted from the saved answer.
Why "they agreed 94% of the time" is the wrong number
The two instruments gave the same answer on 94% of checks. That sounds like strong agreement, and it is not. Most constraints were kept, so two instruments that both say "kept" nearly every time will agree nearly every time, whether or not they can spot a violation.
Cohen's kappa corrects for that: it measures agreement beyond what chance would give. Here it was 0.06, close to none. On the question that matters, which answers broke a rule, the two instruments mostly disagreed. Any evaluation that reports raw agreement on rare events is reporting the base rate.
A second judge reorders the leaderboard
The same responses on two tracks were scored again by a judge from a different provider, Mistral Large 3, across 231 pairs. Two numbers decide whether the first judge can be trusted, and the run publishes both.
| Signal | Value | What it means |
|---|---|---|
| Rank correlation | 0.54 | The order of seven close models depends on the judge |
| Provider spread | 8.57 | One provider's models moved more than others; here DeepSeek V3.2 was scored higher by the second judge |
| Group spread | 2.38 | Criticism of each group of institutions moved by similar amounts, so no sign of a political lean |
| Mean difference | +4.94 | The second judge was more generous overall, which on its own changes nothing |
The run before this one failed the check, and was published anyway
In September, the second judge was Claude Sonnet 4.6, and the check failed: a provider spread of 25.62 and a group spread of 10.84, concentrated in criticism of AI labs. The project published the primary judge's scores with the failure printed beside them, rather than swapping judges until one agreed. A single second judge cannot settle its own bias; publishing the disagreement lets the reader see it.
What model risk teams should take from this
Banks already treat models this way. In the US, the Federal Reserve's SR 11-7 asks for effective challenge and independent validation of models. In the UK, the PRA's SS1/23 sets model risk management principles for banks. In the EU, the AI Act asks providers of high-risk systems to test for accuracy and robustness, and NIST's AI Risk Management Framework makes measurement a core function.
An AI judge is a model too. If it decides whether another model's output is acceptable, it needs the same validation as the model it grades. A score from one unchecked judge is an unvalidated model's opinion.
A checklist for evaluating AI on your own work
The practices that made this run's numbers checkable carry over to any internal evaluation:
- Use two judges from different providers, and publish where they disagree.
- For rare events, report kappa, not raw agreement.
- Count a failure only when both instruments find it, and keep each one's solo findings for review.
- Require a quote, matched word for word against the saved answer, for every failure you report.
- Run questions with known answers first. If the pipeline cannot pass them, believe nothing else.
- Save answers before scoring, never edit a score by hand, and record a hash of each rubric with every score.
Limits and disclosure
Overlooked Bench is maintained by Draimo's founder, a student at Strathmore University in Nairobi and KTH Royal Institute of Technology in Stockholm. Neither university funds, endorses or reviews it. It is a separate open-source project, not a Draimo product: Draimo is not evaluated by it and makes no claim from its rankings. The project has no funding from any lab or anyone else; API costs are paid by the maintainers.
- One response per item per model. A gap between two models on one item is a signal to look at, not a finding.
- Scores come from an LLM judge against published rubrics. A second judge from another provider is run on two tracks, and where they disagree, both numbers are published.
- Seven models, not fourteen: the run stopped adding models when its API budget ran out, and the changelog says so.
- Prompts are in English, so this run says nothing about how these models do in Swedish, German or Swahili.
Questions people ask
- What is LLM as a judge?
- Using a language model to score another model's answers against a rubric, instead of human graders. It is fast and cheap, and it is a model with its own errors and biases, so its scores need checking like any other model's.
- Is one LLM judge enough to evaluate an AI system?
- Not for decisions that matter. In this run, one judge and an independent verifier agreed on one violation out of 14 and 8 flagged, and a second judge reordered the leaderboard. Use at least two instruments from different providers and report where they disagree.
- What is Cohen's kappa, and why use it?
- A measure of agreement between two raters beyond what chance alone would give. When failures are rare, two raters agree most of the time by default, so raw agreement looks high even if neither can find a failure. Kappa does not have that problem.
Sources we checked
- Run 2026-10-22 README: judge, verifier and cross-judge results
- Overlooked Bench methodology
- Federal Reserve SR 11-7: Guidance on Model Risk Management
- PRA SS1/23: Model risk management principles for banks
- Regulation (EU) 2024/1689 (AI Act)
- NIST AI Risk Management Framework
Checked October 25, 2026. Every figure comes from the published run linked above.
Design an evaluation with two judges, calibration questions and a rubric your risk team can sign off.
Design it in Draimo
