Research8 min read
Can AI plan production software under real constraints? We gave seven models 15 tasks
The Draimo team
Short answer
Mostly, yes. In the October 2026 run of Overlooked Bench, seven models planned 15 production engineering tasks with hard constraints: no new services, a frozen database, two developers, a privacy law. Six had no confirmed violation. DeepSeek V3.2 broke one, keeping raw location data past a 30-day limit, and its scores on one feature swung 41 points depending on the privacy law named.
Try it in Draimo
Check your own plan against its constraints
- Here is our design doc. Check it against these hard constraints and quote every line that breaks one: no new services this quarter, the database instance class is frozen, two developers, three weeks.
- We are adding rider location tracking in California. List the CCPA/CPRA questions our design has to answer, and turn our retention policy into yes or no checks we can test.
- Files
- Attach the design doc, runbook or policy and ask Draimo to check it line by line.
- One file per message on every plan; several at once on paid plans
- Agent Mode
- Researches current public information and cites every source, so each claim in a plan or review can be checked.
- Monthly and yearly plans
- Draimo API
- Every model in this benchmark, under its own name, through one OpenAI-compatible key.
- Pay per token, billed in credits
The test: real constraints, scored like arithmetic
Production work is not short of good ideas; it is short of ideas that fit. Each of the 15 tasks gives a team, a budget, a deadline and a set of hard constraints, and states each constraint as a yes or no check. Going over the budget is wrong, the way a wrong sum is wrong, however good the rest of the plan reads.
A violation counts only when the judge and an independent verifier model both find it, and the judge's quote of the offending sentence is matched word for word against the saved response. The tasks are set in the places production software is really built:
| Task | Where | The hard part |
|---|---|---|
| Rate limiting and usage metering | Austin, Nairobi, Lisbon | No new services; same task, three team shapes |
| Rider location tracking | Los Angeles, Berlin, Nairobi | CCPA/CPRA, GDPR or Kenya's Data Protection Act; same feature, three laws |
| MFA on a legacy stack | Columbus | Java 8 on WebSphere, no new vendor |
| int4 to bigint migration | Warsaw | A 410 GB table and a frozen budget |
| Same-day webhook hotfix review | Lagos | No new dependencies |
| Scheduled exports | Amsterdam | Under EUR 600 a month, data stays in the EU |
| Extract one service | Sao Paulo | No new message broker |
| Cut observability spend in half | Pune | CERT-In logging rules still apply |
| M-Pesa reconciliation | Nairobi | A budget of zero |
| Blameless incident review | Cape Town | 15 engineer-days |
| Runbook | Dhaka | For two developers who are not on the ops team |
The results
Claude Sonnet 5 scored highest on the track at 94.2, then Gemini 3.5 Flash at 91.9, GPT-5.4 and GPT-5.4 Mini at 90.8, Gemini 2.5 Pro at 89.9 and Claude Haiku 4.5 at 85.3. None of them had a confirmed violation in 45 constraint checks. DeepSeek V3.2 scored 69.5, and 1 of its 15 plans would have been sent back.
The one plan that broke its constraint
The task: build rider location tracking for a company in Los Angeles under CCPA/CPRA. Constraint C1 says raw precise location may not be kept longer than 30 days anywhere, including the warehouse and backups. GPT-5.4 Mini wrote "Store raw precise location for no more than 30 days" and kept all three constraints. DeepSeek V3.2 wrote "Daily batch job moves 30+ day old raw data to cold storage (compressed, encrypted)", and on the next line deleted it 30 days after that.
That is the shape of most real production mistakes: not a bad idea, but a reasonable one that ignores a line in the brief. Cold storage is still storage, and the plan keeps raw location for about 60 days.
Same feature, three privacy laws
Three of the tasks are identical apart from the jurisdiction: Germany under GDPR, California under CCPA/CPRA, and Kenya under its Data Protection Act. So a gap between them comes from the model, not the question. Most models moved by 11 to 16 points across the three. DeepSeek V3.2 scored 48.8 in Berlin, 42.5 in Los Angeles and 83.8 in Nairobi, a spread of 41.
With one response per law, that is a direction, not a verdict. But it is the kind of gap a team shipping the same product in the US and the EU should test for itself before trusting a model with compliance work.
Same task, three team shapes
The rate-limiting task was set three times, for teams in Nairobi, Austin and Lisbon with different shapes and the same rule against new services. Claude Sonnet 5 scored 100 on all three. The other models moved by 6 to 20 points between them, which is worth knowing if your team does not look like the teams in a model's training data.
What this means for engineering teams
The models are good at this. The failures that remain are the expensive kind, because they look right. What helps:
- Write constraints as yes or no checks, not as background. "No new services this quarter" is a check; "we prefer a lean stack" is not.
- Ask the model to restate each constraint and say how the plan meets it, before it writes the plan.
- Check the plan against the constraints before anyone builds it, and quote the line that breaks one.
- Test the jurisdictions you ship in. A model that handles GDPR well may not handle CCPA the same way.
Why only one violation counts
The LLM judge flagged 14 violations on its own. The independent verifier flagged 8. They agreed on one. Counting only what both find makes the published number a floor, not a census, and it is the reason the number can be trusted. The story of that disagreement is in One AI judge found 14 violations. A second found 8.
Limits and disclosure
Overlooked Bench is maintained by Draimo's founder, a student at Strathmore University in Nairobi and KTH Royal Institute of Technology in Stockholm. Neither university funds, endorses or reviews it. It is a separate open-source project, not a Draimo product: Draimo is not evaluated by it and makes no claim from its rankings. The project has no funding from any lab or anyone else; API costs are paid by the maintainers.
- One response per item per model. A gap between two models on one item is a signal to look at, not a finding.
- Scores come from an LLM judge against published rubrics. A second judge from another provider is run on two tracks, and where they disagree, both numbers are published.
- Seven models, not fourteen: the run stopped adding models when its API budget ran out, and the changelog says so.
- Prompts are in English, so this run says nothing about how these models do in Swedish, German or Swahili.
Questions people ask
- Which AI model writes the best production engineering plans?
- In this run, Claude Sonnet 5 scored highest on the production track at 94.2, followed by Gemini 3.5 Flash at 91.9. Six of seven models had no confirmed constraint violation in 15 tasks. The margins are small and each task was answered once, so test on your own constraints before choosing.
- How is a constraint violation decided?
- Each task lists its hard constraints as yes or no checks. A violation counts only if the LLM judge and an independent verifier model both find it, and the judge's quote of the offending sentence matches the saved response word for word.
- Can I see the full tasks and answers?
- Yes. The 15 tasks, every model's full response and every verdict are public on GitHub, linked at the foot of this page.
Sources we checked
- The 15 production engineering tasks and their constraints
- Run 2026-10-22: responses, verdicts and charts
- Methodology, section 4b: the production track
- California Privacy Protection Agency: CCPA regulations
- Regulation (EU) 2016/679 (GDPR)
Checked October 25, 2026. Every figure comes from the published run linked above.
Paste your design doc and its constraints, and see which lines break them.
Check a plan in Draimo
