The Most Expensive AI Model Came Last

A research team working with a global enterprise built an AI system to investigate supplier-invoice errors, resolve the routine cases and hand the exceptions to a person. The organisation processes more than US$75 billion in supplier invoices each year. The trial compared four model configurations on the same 44 test invoices. ¹

The premium pairing—the strongest reasoning model and a premium document-reading model—resolved 91% without human help. A budget configuration reached 95%. A matched mid-tier configuration resolved all 44. The most expensive option came last.

This is a deliberately small case, not a universal league table. The enterprise is unnamed, the underlying capstone report is not public and the test invoices contained clearly defined errors. The useful finding is not that one model is always better. It is that reputation failed to predict performance on the job being bought.

Every reported failure occurred in duplicate detection. That task required the system to weigh five competing rules at once. The premium model became more cautious around borderline matches; the more constrained model flagged them. Capability that helps on an open-ended problem can introduce hesitation into a tightly defined operational choice.

A second comparison exposed a cost transfer. Swapping in a cheaper document reader did not simply lower the bill. The reasoning agent had to spend more effort cleaning the extracted text, and total computation per invoice rose by roughly 68%. A saving at one step created extra work downstream.

The team also compared a fixed sequence with a more elaborate supervisor design that could review and retry. The supervisor was slower and consumed more computation, but did not resolve one additional case in this test. Complexity may help with genuinely ambiguous exceptions. It should enter because the error set demands it, not because the architecture sounds more advanced.

Model benchmarks, token prices and architecture diagrams are inputs. The business buys a completed outcome: a correct invoice disposition, a useful handoff, an auditable record and an acceptable cost. A component can improve while the full chain gets worse.

This is why procurement cannot end with a preferred-model list. The CFO needs a unit of value. The process owner needs a definition of complete. Risk needs to know which exceptions return to a person and whether that person receives enough context to decide. Technology leaders need to measure the full run after every material model or workflow change.

The US Government Accountability Office found a related management gap across 13 AI acquisitions at four federal agencies: lessons from acquisitions were not being collected systematically. ² Different sector, same operating weakness. If comparison results disappear into a pilot team or supplier presentation, the next purchase begins with the same assumptions.

First, define the completed job in business language. Second, run each viable configuration on the same representative cases. Third, price the whole run, including retries, document processing, oversight and human resolution. Fourth, inspect the exceptions: did the system hand back the right cases with useful context? Fifth, repeat after a meaningful change in model, data or rules.

Do not let an AI system grade itself without a stable check. A new preregistered arXiv study reports that model judges accessed through shared cloud endpoints produced rankings that were not reliably repeatable in the tested setting. ³ It is a preprint, not a final verdict on model-based evaluation, but it reinforces a practical point: validate the measuring instrument before attaching a release or budget gate to it.

This scorecard is intentionally short. The board does not need to choose the model. It needs to require a decision process that makes workflow performance and economic trade-offs visible.

The same-work test travels across markets because every organisation can define an outcome and compare alternatives. The reported winner does not travel unchanged. Invoice formats, languages, tax treatment, credit notes, supplier practices, record quality and labour costs alter both the error set and the business case.

A 4 September Indian retail-banking research report makes the same contextual point from another direction. It builds around local product rules, multilingual conversations, tool order and bank-controlled deployment because generic capability is not enough for the operating environment. ⁴ It is a technical report, not proof of production performance, but the strategic direction is clear: local fit can be part of the system design.

Smaller organisations may not need four elaborate configurations. They can still compare a premium option, a proportionate option and the current human process on a carefully chosen case set. What matters is resisting the shortcut from 'best known model' to 'best business choice.'

Choose one planned model upgrade, agent purchase or renewal. Ask the process owner to name the completed job and assemble a representative set of routine work, difficult work and cases that should be handed back. Include the regional and language conditions the organisation actually serves.

Run at least two viable configurations and the current process. Record completed outcomes, end-to-end cost, elapsed time, human interventions and the quality of each handoff. Do not combine the results into one impressive average that hides a dangerous exception class.

At day 30, approve the configuration that best fits the job, narrow the scope where results are weak, or stop the purchase if no one can show the comparison. The goal is not to buy less capable AI. It is to stop paying for capability the workflow cannot convert into value.