Practice
Write goals that agents can finish.
Most agent failures that get blamed on capability are specification failures. There is a clean natural experiment showing exactly how much of the gap is yours to close.
The same model, twice the score
When OpenAI built SWE-bench Verified, human annotators reviewed the original benchmark and flagged 38.3% of samples for underspecified problem statements. In total 68.3% of samples were filtered out for underspecification, unfair tests, or other issues.
On the cleaned 500-sample set, GPT-4o scored 33.2%, against 16% on the original benchmark. Same model, same weights, same code. More than half of the apparent capability gap was measurement noise caused by unclear tasks.
If a benchmark curated by researchers was 38% underspecified, the brief you typed in thirty seconds is not better.
Letting the agent ask is worth more than better prompting
Ambig-SWE (Vijayvargiya et al., CMU, 2025) tested what happens when agents can resolve ambiguity interactively instead of guessing. Models gained up to 74% performance improvement when allowed to interact and resolve underspecified inputs.
The finding underneath it is the uncomfortable one: models "struggle to distinguish between well-specified and underspecified instructions." An agent does not reliably know when it is missing something. It will proceed confidently on an ambiguous brief and produce something that looks finished.
One success is not reliability
τ-bench (Yao et al., Sierra, ICLR 2025) measures something most demos do not: consistency across repeated runs. State-of-the-art function-calling agents on gpt-4o succeeded on fewer than 50% of tasks, and — the number that matters — pass^8 was below 25% in the retail domain. Fewer than one in four tasks succeeded on all eight independent attempts.
A brief that works once and fails twice is not a working brief. Reliability is a property of the specification as much as the model.
Scope to the horizon that exists
METR's Measuring AI Ability to Complete Long Software Tasks gives a way to size work honestly. The task length that agents complete with 50% reliability has been doubling roughly every 7 months for six years; at the time of that study, Claude 3.7 Sonnet's 50%-reliability time horizon was around 50 minutes of equivalent human work.
The trend line is steep, and the practical instruction that falls out of it is stable regardless of where the frontier sits today: break work into units below the current horizon, with a checkable result at each boundary. That is the same argument for decomposition made in how Modeus routes one task across several AI models, arriving from a different direction.
The four parts of a finishable brief
1. The outcome. Name the artefact, not the activity. "A one-page positioning brief comparing three named competitors" is finishable. "Look into our competitors" is not.
2. The evidence standard. What counts as a source, how many, how recent, and what happens when sources disagree. Without this, the agent picks a standard and you find out what it picked afterwards. The standard we use is in what makes research defensible.
3. The constraints. Format, length, tone, systems it may touch, actions that need approval, and what is out of scope. Constraints stated up front are cheap; constraints discovered in review cost a full rerun.
4. The definition of done. The checks that must pass before the work is handed back — tests run, links opened, figures reconciled, coverage reported. This is the part most briefs omit entirely, and the part that determines whether "finished" means anything.
Anthropic's guidance on building effective agents lands in the same place: agents suit tasks with "clear success criteria" that enable feedback loops, and tool definitions deserve "just as much prompt engineering attention as your overall prompts."
A useful test before sending: could a competent stranger tell whether the result is acceptable, using only what you wrote? If not, the agent cannot either — and per Ambig-SWE, it probably will not tell you.
Sources
Every figure in this article was read from the source below on 2 August 2026. Where a source carries no publication date, an access date is given instead.
- Introducing SWE-bench Verified38.3% of samples flagged underspecified; GPT-4o 16% to 33.2% on the cleaned set.
- Interactive Agents to Overcome Ambiguity in Software EngineeringUp to 74% improvement when agents can resolve underspecified inputs interactively.
- τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainsgpt-4o below 50% task success; pass^8 under 25% in retail.
- Measuring AI Ability to Complete Long Software Tasks50%-reliability task length doubling approximately every 7 months.
- Building Effective AI AgentsClear success criteria as a precondition for agentic work.