Ask Which AI Errors the Average Hides

Ask the team to show what happens without the proposed AI system. Depending on the task, that comparison might be the existing process, a straightforward rule or a person following the current checklist. A sophisticated model should not receive credit for solving a problem that a simpler method handles just as well.

Keep the comparison fair. Use the same cases, the same definition of success and the same deadline. Include the work needed to check and correct the output. A result that looks better before review may be less useful once that work is included.

Do not make the baseline artificially weak. If the current team has access to useful information, compare the proposed workflow with that reality. Equally, do not assume the current process is safe merely because it is familiar. Document its mistakes too.

The funding question becomes concrete: what improves, for whom, and at what additional cost? If the answer is not yet clear, fund a bounded comparison rather than describe the pilot as proven.

Recognising a situation, explaining it and choosing an action are different jobs. A test of one should not silently authorise the next. Write down the actual decision the system will influence and what evidence supports each step towards it.

For a fictional warehouse example, spotting a damaged carton is different from deciding whether its contents can be dispatched. The second decision may require information that the photograph cannot supply. A fluent explanation should not be allowed to fill that gap with an assumption.

Choose a small set of errors that would change the approval decision. Include missed problems, unnecessary alarms and cases where the system should ask for help. The relative cost depends on the workflow; there is no universal acceptable accuracy percentage.

Show the number of examples behind each result. If only a handful of consequential cases were tested, say so. An empty row is missing evidence, not a perfect result. Have someone with operating knowledge review disputed labels before treating them as unquestionable truth.

A separate October preprint, AlphaDiana, compares agent configurations on specified reasoning benchmarks. Adding tools, skills or retained history did not consistently improve correctness in those tests. It is not a production reliability study, but it supports a useful question: which complete configuration produced the result being presented? ²

Record the model, instructions, available tools, input format and review process used in your own trial. If one changes, decide which tests need to run again. Do not carry an approval across a material change simply because the product name remains the same.

Also specify what happens outside the tested conditions. A system might return work to a person, decline a request or stay in an advisory role. That fallback needs an owner and enough capacity to handle the expected workload. A theoretical escalation path is not an operating safeguard.

The Bangladesh case should keep its geography. It is not a proxy for Asia-Pacific, and no result here establishes comparative performance in North America, Europe, Africa, the Middle East or Latin America. The question that travels is whether the test resembles the intended use.

For a multinational organisation, compare sites rather than assume a country label explains them. Language, equipment, document conventions, connectivity and the availability of experienced reviewers may change the task. Two sites in one country can differ more than two across a border.

Ask local staff and affected users which examples the central team would overlook. Obtain suitable rights to use test material and minimise personal information. Local privacy and sector requirements need their own review; a benchmark result cannot stand in for it.

Make the supplier's evidence gaps visible in procurement. Specify who will produce local validation, who pays for it and what happens if the proposed use is unsupported. Avoid turning the customer's uncertainty into an unqualified claim of global readiness.

Ask for one page containing the intended decision, a fair baseline, the consequential errors, the conditions tested and the fallback. Attach a few reviewed examples so the discussion stays connected to actual work.

Approve only the responsibility that the evidence supports. A useful assistant may still be unsuitable for independent action. Where an important question remains unanswered, give the next test a clear owner and completion condition before expanding access.