When New Evidence Must Reopen an AI Decision

A preprint submitted on 14 October examines what happens when earlier reasoning conflicts with later evidence. Across three model families and two English question-answering benchmarks, the way the answer was requested affected whether it followed the updated evidence. The experiment used constructed conflicts; it is not a measurement of commercial systems or enterprise error rates. ¹

There is also a concrete reminder to track the version of evidence being used. A digital-twin research paper revised on 15 October explicitly corrected its case study and all reported measurements. That is a disclosed revision, not evidence of misconduct. It means an assessment relying on the earlier measurements would need to check whether its conclusion still holds. ²

These findings do not establish that a particular supplier's product is unsafe, or that separating every AI task into more steps will solve the problem. They identify questions worth testing. What previous reasoning is retained? Which evidence is current? Can the workflow identify an important contradiction rather than quietly blending old and new material?

A useful test should permit the conclusion to remain unchanged. New evidence may strengthen an existing decision, be irrelevant to its scope or be too uncertain to displace the original basis. The objective is justified reconsideration, not automatic reversal. Record the reason for retaining a decision as carefully as the reason for changing it.

Consider a fictional supplier review. An AI tool summarises a supplier's service statement, and a manager approves a limited rollout. Later, the supplier corrects which locations are supported. Updating the statement in the document store is the first outcome. Producing an accurate revised summary is the second. Checking whether the approved rollout includes an unsupported location is the third.

A team that proves only the first two has not demonstrated that the business decision is current. The rollout owner needs to know what changed, which part of the approval relied on it and whether work should continue while the issue is checked. A generic notification that a document has been updated may not provide enough information.

Build a small decision record around consequential uses. It should identify the decision, accountable owner, source and version, material assumptions, intended scope and review trigger. Link to supporting evidence through existing access controls. The purpose is not to copy every confidential document into a new database or collect employees' private conversations.

Start manually if necessary. A simple register for a few important decisions can expose missing ownership before investment in automation. Where a system can map those links reliably, use it to suggest affected decisions. Keep the authority to change a contract, withdraw access or reverse an operational instruction with the people already authorised to do so.

On 15 October, Anthropic described deeper access for six METR evaluators and different access tiers for other partners. Its clarified terms say it cannot veto METR's publications, but can require redaction of predefined categories and decides whether information falls within them. Other arrangements have narrower access and publication rights. This is a voluntary supplier arrangement, not a universal audit standard. ³

For buyers, the practical question is not whether a supplier uses the word independent. Ask what the evaluator can inspect, what it cannot inspect, what it can publish and how material findings reach customers. An assessment with a clear boundary can be useful. An unspecified assurance badge cannot answer which of your decisions it supports.

Apply the same discipline internally. A central AI team may know that a model, tool or evidence source has changed while the operating team sees only the same product name. Give material changes a named recipient, not merely a shared inbox. Agree what the recipient must acknowledge and when unresolved matters escalate.

There are legitimate limits. Customer confidentiality, security and intellectual property can restrict disclosure. Seek enough information to assess your exposure without demanding unnecessary sensitive detail. If the supplier cannot provide evidence needed for a consequential use, narrow that use or retain an alternative. Absence of public detail is neither proof of safety nor proof of failure.

NIST's 16 October practices for super-intelligence training and evaluation include recovery to a known-good state and documentation of recovery decisions and residual risk. The document is voluntary and aimed at that specialised setting; it is not a new compliance requirement for ordinary business users. The relevant management analogy is that restoration and permission to resume are distinct judgements. ⁴

For an enterprise workflow, classify the change before choosing the response. A correction to presentation may need no decision review. A changed limitation, permission, price assumption or result may require the relevant owner to reassess reliance. Credible evidence of immediate harm may justify a temporary pause under existing incident procedures. Those thresholds should be agreed before a busy team faces an ambiguous alert.

Set a response time according to consequence, not one arbitrary deadline for all AI work. Identify who can pause an affected activity, who checks the new evidence and who can authorise continuation. A decision to continue with uncertainty should say what remains unknown and what exposure is accepted. Avoid making the technology team silently carry a business risk it cannot authorise.

Do not confuse speed with completion. A source may refresh in minutes while an affected customer communication remains uncorrected for days. Track the time from a material change being recognised to affected decisions being reviewed. Also retain the number still unresolved and any interim protection. These are proposed operating measures, not published benchmarks or promised performance gains.

The principle travels because organisations everywhere make decisions using information that can change. The implementation should not assume that every office has the same suppliers, permissions, connectivity, language support or legal obligations. A shared model name is not evidence of an identical operating environment.

China's 16 October technology-export announcement illustrates the importance of separating kinds of change. It combines consultation on a proposed catalogue revision with temporary registration requirements for specified technologies. A cross-border transaction needs its own scope assessment; a proposal and an active requirement cannot be placed in the same category. This is not a blanket conclusion about Chinese AI services. ⁵

For operations in Europe, the United Kingdom or North America, ask local specialists which existing records, contracts and sector duties constrain retention and notification. Do the equivalent in Asia-Pacific, Latin America, Africa and the Middle East rather than copying a headquarters checklist. This article establishes no common legal deadline or right to transfer underlying evidence across borders.

Language and institutional capacity matter as well. If the supporting research tests English questions, do not treat it as validation of a multilingual workflow. If an office has limited review capacity or intermittent access, a smaller, clearly owned decision register may be more useful than a complex alert system. We lack comparative evidence on correction performance across these regions; that gap calls for local testing, not regional rankings.

Choose one consequential workflow with a willing operating owner. Map a small set of decisions and their evidence, then introduce a clearly labelled fictional correction in a test environment. Verify that the workflow finds the relevant decisions without disturbing unrelated ones. Check both the revised answer and the notification received by the decision owner.

Run a second case where the new information does not change the conclusion. The owner should be able to retain the decision with a reason. Include one ambiguous case that requires human judgement. The test should reveal whether the process can distinguish a material change from noise, not merely whether it generates more alerts.

Bring the board or executive sponsor a short result: what was detected, what was missed, who reviewed it, what remained unresolved and what the next expansion would require. Build the missing links before buying a larger reporting layer. Watch research on changing evidence and configuration, but do not treat one benchmark improvement as proof that your correction process works.

The immediate decision is not to reopen the whole AI portfolio. It is to refuse to call a consequential workflow complete until someone can show how its decisions stay connected to the evidence they depend on. An approved answer should not become an orphan when the facts change.