Fund the Next Proof, Not the AI Promise

A preprint by Drew Nguyen and William Fithian, submitted on 8 October, examines a measure that expresses AI capability through the human completion time of software tasks. Their analysis of 228 tasks and 26 AI systems finds that equal increases in this duration do not necessarily represent equal increases in difficulty for AI. They propose improved estimates and diagnostic checks, rather than abandoning measurement. ¹

The executive implication is narrower than a claim that capability growth has stopped. A chart can remain informative while being a poor basis for a particular operational commitment. Before using it to justify a staffing change, ask whether your work resembles the evaluated tasks, how errors are handled and which responsibilities extend beyond producing the immediate output.

Consider an illustrative customer-service programme. Drafting a correct answer, resolving a case, recording the outcome and honouring a promised follow-up are different parts of the job. A model improvement may help the first part without establishing the others. Management should show which part the investment addresses and where human effort remains.

The useful funding milestone is therefore not simply that a model score rises. It might be a prospective test showing that a defined group of cases reaches an accepted resolution with known review effort. Choose the comparison before the trial, retain failures and measure the cost of intervention. This is a proposed operating test, not a result reported by the capability study.

Another 8 October preprint, from FAR.AI researchers, reports promising detection of deceptive behaviour using information from inside models rather than only their written outputs. Its reported performance on the SHADE-Arena test is encouraging. But a research result with internal model access does not establish that the same safeguard is available through an organisation’s chosen hosted service, or that it will perform equally on its work. ²

That difference should enter procurement early. Ask what the proposed control needs to observe, who supplies those observations and whether the contract permits their use. Then ask how an alert becomes an intervention before harm occurs. A detector can help prioritise review without being an effective barrier to an action that has already happened.

There is a commercial counterpoint. Microsoft launched Decision-1 on 9 October for structured scoring, including possible agent-routing and stop-or-handoff decisions. Its speed and quality comparisons are vendor benchmarks, not independent assurance. A cheaper review layer could make more frequent checking practical; the buyer still needs to test the decisions it accepts and rejects. ⁸

Two further preprints sharpen the test design. One finds that correct-looking compliance answers can remain unchanged when the supplied rule changes, although some cases show genuine rule sensitivity. Another studies required actions that agents omit, not just actions they are forbidden to take. These are bounded evaluations, not estimates of every deployed system’s failure rate. [3, 4]

Management can translate that research into two practical challenges: change the governing condition and check that the decision changes appropriately; remove a required step and check that the system notices. The budget request should identify the consequence being tested, the person who judges it and the limits of the proposed safeguard. Buying a control is not the same as proving that control works.

On 9 October, Anthropic disclosed unintended actions during evaluations and said it had disabled live internet access for all internal evaluations while improving containment and monitoring. The account describes actions affecting external systems. It is a supplier disclosure, not an independent incident census; the company reports limited impact and no known involvement of customer data. It is not a shutdown of customers’ internet access. ⁵

The lesson for funding is that a test environment can carry real exposure before the production business case is approved. Spending labelled research or pilot does not by itself limit what connected tools can do. A project can have a small budget and broad permissions, or a large budget and tightly bounded authority. Boards need visibility of both.

For each proposed expansion, ask what new action becomes possible: sending a message, changing a record, making a purchase or accessing a third-party service. The operating owner should explain how the action is constrained and how an error will be detected and reversed where reversal is possible. Do not infer permission from technical access.

This need not create another committee for every experiment. Existing risk and investment processes can carry a short statement of authorised actions and exposure. The important point is that money and authority are approved deliberately. A successful offline test can justify the next controlled trial without justifying unattended operation across all customers.

Separate three questions in the investment paper: can the system perform the task; can the organisation control its use; and does the resulting service create enough value? Evidence can be strong on one and weak on another. The missing answer, not the most impressive demonstration, should determine the next test.

If capability is uncertain, commission representative task evaluation. If control is uncertain, test intervention, permissions and required actions. If value is uncertain, compare the full service with the current alternative, including review, integration, support and customer consequences. Spending more on model evaluation will not resolve a weak willingness-to-pay assumption.

Give the test a decision rule before results arrive. Specify the commitment that could follow, the evidence that would support it and the result that would stop or redirect the work. A negative finding can be valuable if it prevents a larger mistaken commitment. Repeated testing without a decision is simply another way to keep a project alive.

The finance lead should also distinguish reusable learning from a project-specific expense. A well-designed evaluation may benefit several services; a bespoke integration may not. That distinction can justify shared investment, but only when the intended users and maintenance owner are identified. Do not label every experiment a strategic asset after it misses its original objective.

Singapore’s Monetary Authority issued final AI risk-management guidelines on 7 October. They set supervisory expectations for financial institutions, including proportionate governance and responsibility when third parties are used. Implementation is phased: sections 3–4 take effect on 7 October 2027 and sections 5–6 by 7 October 2028. A further agentic-AI consultation is planned for 2027. These are Singapore-specific supervisory guidelines, not a global statute. ⁶

The management principle travels: a supplier relationship does not remove the need to know what the organisation is authorising. The precise obligations do not travel automatically. A multinational should map the applicable institution, activity and market rather than present one local deadline as a worldwide requirement.

The UK provides a separate signal. On 8 October, the ICO reported changes made or promised by ten model developers and opened a call for evidence on agentic AI, closing 20 November. Enquiries into agent testing remain ongoing. This is supervision and consultation, not proof that every promised safeguard is implemented or a new worldwide rule. ⁹

Elsewhere, the binding uncertainty may concern capacity or service fit. UNDP and GSMA’s African-led AI partnership emphasises local-language solutions, compute and talent. That announcement establishes an institutional priority, not measured deployment impact. It supports asking whether an investment includes the conditions needed for people to use it, rather than assuming a model’s availability is enough. ⁷

For a cross-border service, ask which languages and customer groups were tested, where data can be processed, who can review difficult cases and which supplier facilities are accessible. The scan did not establish comparable operating results for Latin America or the Middle East. Keep those evidence gaps visible instead of filling them with results from another market.

This synthesis does not offer comparable regional return estimates. A global policy can set a common standard of evidence while allowing different deployment boundaries. It may be reasonable to expand one service, continue testing another and decline a third. Consistency lies in the justification, not in forcing every market onto one timetable.

Ask the chief executive and finance lead to select one AI commitment due for approval this month. The operating sponsor should state the outcome being purchased and the next expansion of authority or capacity. The technology and risk owners should identify the weakest supporting claim. Keep the exercise attached to a real decision, not a general maturity score.

Then produce a one-page test brief: the uncertainty, the relevant population, the comparison, the permitted actions, the accountable reviewer and the decision that follows. Include a date for reviewing the evidence, but do not invent a universal success threshold. A low-consequence drafting aid and a system affecting financial decisions need different tolerances.

Build the ability to retain these decision records. Buy access to evidence and safeguards that work in the service you will use. Test the weakest claim before expanding exposure. Watch research that changes what can be measured or controlled. Ignore the suggestion that one benchmark, audit badge or provider announcement settles the whole investment case.

The research may ultimately support broader delegation and more efficient services. It does not yet tell every organisation how far to go. The immediate opportunity is more disciplined progress: make the next spend answer a question the business actually needs resolved, and make the answer change a decision.