Ask What Your AI Upgrade Actually Buys

A new preprint from Amazon and New York University examines changes to the software around a model: its instructions, tools and task-management logic. Across 25 experimental settings, 21 showed statistically significant gains on the work used for optimisation. In 14 of those 21, detected test-specific shortcuts and additional computing together explained more than half the gain. ¹

The remaining gain is an upper bound on possible transferable improvement, not proof of it. Coding experiments showed more promising residual gains than some other tasks. The study was submitted on 15 October and appeared in the 19 October arXiv batch; it is not peer-reviewed production evidence. ¹

This is not a verdict on all agents or an instruction to reject automation. It is a reason to ask a more precise buying question. If the old design improves when given more attempts, how much of the new offer's advantage still needs its additional software, service fee or implementation work?

OpenAI's 14 October EU analysis reports growing activity in its agentic products, which can carry out multi-step tasks. Its intensity comparisons use output tokens per active user among sampled Enterprise and Business accounts. Tokens are pieces of generated text, not completed business jobs. The report does not establish that the most intensive users earn the highest returns. ²

Payment data tells a different part of the story. Capital on Tap's 12 October report found that 14.5% of active UK businesses in its card dataset paid identified AI providers during the third quarter. That measure excludes free plans, app-store subscriptions and invoiced contracts. It is neither total UK adoption nor a global market estimate. ³

These signals show why purchase discipline matters without proving that spending is wasteful. A token count records activity; a card payment records a purchase. The investment case needs a further link to a useful outcome. Avoid using growth in either measure to justify an upgrade whose incremental contribution has not been tested.

First, run the current design under its normal limits. Second, run that same design with a computing allowance comparable to the proposed upgrade. Third, run the upgrade. Use the same authorised cases, completion rules and permission boundaries. Record failed attempts as well as successful ones. This is a proposed business comparison, not a certification method.

Match what matters to the purchase. Similar token totals may not mean similar money, elapsed time or access to tools. Parallel attempts can finish sooner but consume more capacity. A slower process may be affordable for overnight work and useless for a live customer request. Report those differences rather than hiding them inside one score.

Then reserve a second set of work that the supplier and tuning team have not used to improve the system. Include relevant variations in format, language and difficulty. Do not keep adjusting the product against this set and continue calling it unseen. Where results drive a material commitment, separate the people improving the system from those deciding whether it passes.

An upgrade that wins both comparisons has a stronger case, not a guarantee. An upgrade that only wins with more computing may still be worth buying. Ask whether that package is cheaper or easier to operate than adding the capacity yourself. An upgrade whose advantage disappears outside the demonstration needs a narrower claim and a smaller commitment.

Use a small, explicit business case. How many additional acceptable jobs does the upgrade finish? What extra computing, licences, integration, monitoring and human review does it require? Which delays or unacceptable errors remain? Divide the incremental cost by the incremental useful outcomes only when the units and comparison period genuinely match.

Do not turn a research percentage into a savings forecast. A benchmark gain is not a percentage reduction in labour, and test-specific behaviour is not automatically fraud. Narrow, predictable routines can be useful when their operating boundary is deliberate. The problem is selling a narrow advantage as a broad capability without evidence.

Require the budget owner to name what would justify renewal. That might be lower cost at the same accepted quality, more completed work within the same deadline, or a new service that was previously impractical. Stop or narrow the premium when that claim fails. Keep safety and permission limits as conditions, not costs that a higher average score can cancel.

The buying method travels; the winning option may not. The UK payment sample and EU product telemetry do not establish demand or returns in North America, Asia-Pacific, Latin America, Africa or the Middle East. Nor do they validate a multilingual service. Use local customer work and the actual available product configuration.

Computing prices, network delays, payment arrangements and review capacity can change the answer. So can rules governing where data may be processed and who may inspect test cases. A team with limited connectivity may value a simpler local routine; another may justify remote parallel processing. These are conditions to measure, not regional stereotypes or legal conclusions.

Start with one upcoming renewal or upgrade, not a new enterprise-wide programme. Ask its sponsor to show the three-way comparison and the unfamiliar-work result before the commitment hardens. The aim is not less ambitious AI. It is a clear account of what the next payment actually buys.