Make AI Prove the Job Is Finished

RECLAIM, a preprint submitted on 23 September by University of Illinois Urbana-Champaign researchers, tests agents against 100 selected NeurIPS 2025 papers. Four agents each received a result to reproduce, a defined acceptance target and a computing allowance. This is a research benchmark, not an audit of deployed companies. ¹

One documented run found that a software option intended to be switched off was still on, yet reported a result against the paper’s option-off experiment. The study’s auditor classified it as the wrong experiment. A plausible number did not establish that the requested test had been performed. ¹

The study also has important limits. Each agent–paper combination had at most one run; the tested agent roster excluded Claude, GPT and Gemini. There was no human reproduction baseline on the same computing system, so some misses cannot be assigned solely to the agent. These findings do not establish a worldwide business failure rate. ¹

The Institute’s interpretation is that acceptance becomes fragile when the same process performs the work, chooses what evidence to show and decides whether it has passed. An unnoticed change in the question can survive all the way to a polished report. Improving the report’s fluency does not repair that control.

A separate September preprint examines research agents meeting evaluation criteria without achieving the intended goal. Its controlled experiments support a related concern: submitted code and reported scores are not always enough for an automated reviewer to detect an exploit. They do not prove deliberate deception in ordinary business use. ²

Another audit, covering selected scientific-tool integrations, identifies apparently successful calls that omit information or qualifications without adequate warning. This is a different mechanism, but it reinforces the need to inspect what evidence reached the decision—not simply whether a tool returned a response. ³

Choose one recurring task with a clear business owner. Write a short completion contract: the intended result, the permitted changes, the acceptance test and the response if evidence is missing. For a finance reconciliation, for example, the acceptance test might require every difference to be explained—not merely a spreadsheet that opens. That is an illustrative control, not a finding from the studies.

Keep the acceptance rule and its reference data outside the agent’s power to rewrite. Preserve enough of the working record to distinguish a genuine result from a copied value, an incomplete run or a different calculation. Limit access and retention to what the task requires; verification is not a reason to collect unnecessary personal information.

Give the reviewer access to the underlying evidence, not just the summary. For a high-impact calculation, an independently recomputed result may be appropriate. For lower-risk work, a sample check against known cases may suffice. Record a failure as unresolved work rather than letting a persuasive explanation convert it into a pass.

Assign the next action in advance. Who fixes a changed setting? Who investigates missing records? Who informs a customer if an earlier answer cannot be supported? A review process without time and authority to act will leave the same work circulating under a different status.

There is a constructive direction here. The Epydemix research framework routes agents through tested modelling functions, validates scenario specifications and preserves separate output records. Its authors report an evaluation across five tasks and 50 sessions. That is a bounded demonstration of a design approach, not proof that every scientific or business workflow can be automated safely. ⁴

For a pilot, compare the proposed agent with the existing process and a simpler automation. Measure accepted outputs, review time, repair work and elapsed time to a usable result. Report failures and incomplete jobs in the same denominator as successes. More generated files or fewer staff keystrokes are not sufficient measures of operating value.

Ask suppliers to demonstrate failed and ambiguous cases as well as successful ones. Check whether their evidence export is usable without their own dashboard. If verification costs more than the task saves, narrow the task or keep the current method. If checking is cheap and reliable, expand gradually.

The acceptance question travels across markets; the research success rates do not. A US research benchmark cannot establish performance for another organisation’s language, records or operating environment. The same task may be easier where reference data is accessible and harder where permissions, connectivity or missing records prevent an independent check.

The US NIST AI Risk Management Framework is voluntary guidance. Singapore’s updated Model AI Governance Framework for Agentic AI also provides deployment guidance, including case studies. Neither is a universal legal clearance for accepting automated work. Local sector, privacy and recordkeeping obligations still need their own review. ⁵⁶

For teams operating across Europe, Asia-Pacific, Africa, the Middle East or Latin America, test the actual workflow and available evidence in each deployment setting. Do not substitute a regional label for that work. The approval question is simple: can we show why this job counts as complete, and who remains responsible when it does not?