Check What Must Be True Before AI Acts

A new US preprint explores deriving rules from software-tool documentation, refining them against recorded interactions and checking proposed actions. Its repair mode changes an argument only when the replacement is uniquely determined; its warning mode supplies feedback after execution. That distinction matters: the study does not establish a universal pre-action blocking system. ¹

On 417 simulated AppWorld tasks, one model's success rose from 45.1% to 50.5% under a restricted action budget. Results varied: improvements were smaller in the standard setting, and another model declined on a separate airline-and-retail test. Rules can be wrong, hidden state can escape checking, and the checks add delay. The paper was submitted on 18 October and listed on 20 October; it is not peer-reviewed production evidence. ¹

The useful direction is narrower than a claim that agents are now dependable. Some process knowledge can be represented outside the model's next answer and checked consistently. Whether that reduces operational risk depends on the quality of the rules, what the system can observe and what the organisation permits it to do.

First ask whether the action is technically valid. Does the identifier come from the required lookup? Is the requested quantity within a verified limit? Then ask whether it is authorised. A valid record and a working connection do not establish permission to change it. Finally, ask whether the action serves the customer's or organisation's purpose. Passing the first two checks does not answer the third.

A separate Hong Kong research preprint illustrates the outcome problem. ServiceBench reconstructs 2,500 complaint scenarios across five industries. Simulated agents often offered compensation before collecting enough information. The work uses English scenarios associated mainly with major US brands and model-based evaluation; it is not evidence of actual customer retention, labour savings or performance across languages. ²

A refund can therefore be executable without being the right resolution. A workflow can contain valid steps while omitting the step that matters. Keep a final outcome check and an accessible human route. Do not let a cleaner error log become the sole measure of whether the service worked.

Choose a bounded workflow and document its current failures. Identify a few prerequisites that operations staff already understand. Run proposed checks alongside the existing process without giving them control of live changes. Compare their judgements with reviewed cases, including cases that should pass. This observation stage is an Institute recommendation, not a protocol validated by the new study.

Look for false blocks as carefully as missed errors. A rule inferred from a handful of successful examples may reject a legitimate exception. An old procedure may encode an obsolete restriction. Give the process owner a way to inspect the evidence, approve changes and retire rules. Record the version used for each decision so that a later investigation can reconstruct what happened.

Only then consider enforcement. Missing evidence should trigger a defined stop, fresh lookup or human review, rather than an invented value. Automatic correction belongs in an even narrower category: the replacement must be unambiguous, authorised and consistent with the intended task. Prefer reversible cases, and record both the proposed and executed values. Do not use a technical repair mechanism to make a new commercial choice.

Measure the whole workflow. Count acceptable completed jobs, prevented errors, legitimate actions blocked, additional waiting time and human interventions. Include the work needed to maintain rules as tools and processes change. A system that avoids failed calls but creates an unmanageable review queue has shifted the burden rather than removed it.

Ask suppliers which conditions they can actually verify and which depend on assumptions. Can the system tell that information is stale? What happens when another service changes a record between checking and acting? Which actions are outside the record it observes? Require concrete answers for the proposed workflow, not an assurance based on the product's general architecture.

The acceptance decision should belong jointly to the operating owner and the relevant risk or control owner. Set a narrow expansion condition before the trial starts. For example, require fewer avoidable errors without unacceptable false blocks or delay on reviewed work. The threshold must reflect the task's consequences; this briefing supplies no universal pass rate.

The operating question travels across markets, but neither study establishes global production performance. North American synthetic applications and English complaint simulations do not validate multilingual services in Asia-Pacific, Europe, Latin America, Africa or the Middle East. Local permissions, records, customer expectations and available review capacity can change which checks are useful.

Regulation also needs its own interpretation. Singapore's October AI risk-management guidelines set supervisory expectations for financial institutions, including lifecycle controls and third-party assurance. They are not a universal legal rule or a certification of this research method. ³ Elsewhere, determine applicable obligations with the responsible local specialists rather than copying another jurisdiction's checklist.

For the next operating review, ask one team to demonstrate the difference between a valid action, an authorised action and a useful outcome. If it cannot show where each is checked, hold the expansion of authority. Invest first in the missing evidence and ownership. More autonomy should follow a clearer operating contract, not merely a more fluent demonstration.