Engineering

Verification is part of the deliverable.

When researchers re-examined a coding benchmark's successful patches, roughly a third had the answer leaked in the issue text and roughly a third passed only because the tests were weak. Removing both cut the headline score by two thirds.

By The Modeus team 2 August 2026 6 min read

A rocky path with clear checkpoints marked along its length, used as the article's opening artwork.

Passing is not the same as working

SWE-Bench+ (Aleithan et al., York University, 2024) audited what "resolved" actually meant on a widely-cited coding benchmark. 32.67% of successful patches involved solution leakage — the fix was present in the issue report or its comments. A further 31.08% passed only because of weak test cases.

With both removed, SWE-Agent with GPT-4 fell from a 12.47% resolution rate to 3.97%. The work had not changed. The verification had.

This is the general shape of the problem: a green check mark is evidence about the checking apparatus at least as much as about the code.

Models cannot reliably check their own reasoning

The obvious fix — ask the model to review its own output — has been measured, and it makes things worse. Large Language Models Cannot Self-Correct Reasoning Yet (Huang et al., Google DeepMind and UIUC, ICLR 2024) found that without external feedback, intrinsic self-correction degrades accuracy.

GPT-4 on GSM8K went 95.5% → 91.5% → 89.0% across two rounds of self-correction. GPT-3.5 on CommonSenseQA collapsed from 75.8% to 38.1% after a single round. The model talks itself out of correct answers.

"Please double-check your work" is not a verification step. It is another generation step with a different prompt.

External signal is what changes the outcome

SWT-Bench (Mündler et al., ETH Zurich, NeurIPS 2024) tested the useful version: generate tests, then require that a test fails before the fix and passes after it. Used as a filter, SWE-Agent achieved over twice the precision on fixes that passed self-generated tests meeting that condition.

The fail-then-pass requirement is what makes it real. A test that passes both before and after the change proves only that it does not exercise the change.

Meta's TestGen-LLM deployment (Alshahwan et al., ACM FSE 2024) shows what the filter costs in practice. Of LLM-generated test cases, 75% built correctly, 57% passed reliably, and only 25% increased coverage. But of the survivors that cleared the pipeline, 73% were accepted for production by Meta engineers.

Three quarters of the generated output was discarded, and the quarter that survived was good enough to ship. That ratio is the argument for automated filtering rather than automated trust.

What counts as exercised

The rule we work to is that a deliverable is unfinished until something outside the generating model has been made to agree with it:

Code is unfinished until the project's quality gates have run and the actual output has been inspected — not until it compiles.
A route or page is unfinished until it has been opened and the rendered result checked, because a build that succeeds says nothing about what a user sees.
A claim is unfinished until its source has been opened and read back. Given that only 51.5% of AI-generated sentences are fully supported by their citations, this has to be per claim — see what makes research defensible.
A fix is unfinished until a check exists that would have failed before it.

A software workflow: trace before changing, reproduce the failure, fix the root cause, exercise the result. — The software development workflow. Reproduce comes before fix, and exercise comes before handoff.

Verified work is the only work worth reusing

There is a second reason to insist on this, beyond correctness today. Reuse amplifies whatever you store. An unverified solution reused across ten later tasks is a defect with a distribution channel.

A billing retry example across four stages: the team solves the first failure, Modeus preserves the reusable knowledge, a related retry issue appears months later, and the user can inspect the match. — Active Memory stores the approved root cause, the constraint, the regression test, and the smallest reliable fix — and shows which memory was used and why.

The inspectability in that last step is not a nicety. If you cannot see which prior decision was reused and why it was judged relevant, you cannot tell whether a later failure came from the new work or from something the system learned incorrectly months ago.

Verification is what makes a result safe to store. Storage is what makes verification pay for itself more than once.

Sources

Every figure in this article was read from the source below on 2 August 2026. Where a source carries no publication date, an access date is given instead.

SWE-Bench+: Enhanced Coding Benchmark for LLMsAleithan et al., arXiv:2410.06992 · 9 October 202432.67% solution leakage, 31.08% weak tests; SWE-Agent+GPT-4 from 12.47% to 3.97%.
Large Language Models Cannot Self-Correct Reasoning YetHuang et al., ICLR 2024 / arXiv:2310.01798 · 3 October 2023GPT-4 on GSM8K 95.5% to 89.0% over two self-correction rounds.
SWT-Bench: Testing and Validating Real-World Bug-Fixes with Code AgentsMündler et al., NeurIPS 2024 / arXiv:2406.12952 · 18 June 2024Over twice the precision on fixes passing self-generated fail-then-pass tests.
Automated Unit Test Improvement using Large Language Models at MetaAlshahwan et al., ACM FSE 2024 / arXiv:2402.09171 · 14 February 202475% built, 57% passed reliably, 25% raised coverage; 73% of survivors accepted for production.
SWE-bench VerifiedSWE-bench, Princeton NLP and Stanford with OpenAI · Accessed 2 August 2026A human-validated subset of 500 instances — the benchmark itself required re-verification.