Treat AI Lessons as Proposed Changes

In the skill-adaptation study, researchers tested instructions supplied to agents across seven models and three benchmarks. On a deliberately selected set of 103 task–instruction pairs, revisions informed by execution feedback improved final scores relative to a comparison loop without the additional guidance. The model's underlying parameters were unchanged. ¹

This offers a practical direction: an organisation may improve how an agent follows a procedure without commissioning a newly trained model. The object being changed is the working guidance around the model. Procurement should ask which parts of that guidance the supplier can rewrite, what evidence is retained and whether customers can inspect or constrain the changes.

The study does not establish a universal improvement rate. The selected tasks were chosen because models differed in their ability to use the instructions; across the broader task set, the pattern was less pronounced. The five-round experiments ran each round once, and code release was promised after review. Retrieval and choosing among many skills were outside scope. ¹

A sensible trial therefore starts with a repeatable job and an observable result. Producing a draft document or checking a non-sensitive test file is easier to evaluate than an open-ended claim that the agent has become wiser. Keep the original instructions available. Compare the change against them, rather than assuming that a newer instruction is a better one.

A separate registered preprint studied 256 synthetic settings in which an agent chose between two tools. After a chance failure, an AI-written lesson sometimes shifted the next choice away from the tool with the better recorded reliability. The reader could still see the underlying log. This tests a possible failure mode, not its frequency in businesses. ²

Important limits remain. The agent named a tool without executing it; only one stored lesson was tested. Results differed across model endpoints, and ambiguity about whether a failure signalled changed reliability mattered. The study was not independently replicated, and its author reports no external human review. It cannot establish that persistent memory is generally harmful. ²

Imagine an assistant checking stock through a normally dependable service. One request times out. The assistant saves a lesson saying to use a less dependable source first. Tomorrow's answer may arrive more readily yet be less useful to the person deciding what to order. That is an illustration of the operating concern, not a reported deployment incident.

The opposite mistake also matters. A failed request might reveal a genuine change: the service could have moved, lost access or begun returning stale information. A control that rejects every lesson from a failure would preserve old mistakes. The question is what additional evidence distinguishes a temporary interruption from a reason to change the procedure.

Keep the event record separate from the lesson. The record should say what was requested, which version ran, what happened and what remains unknown. The lesson should state the proposed change and why the event supports it. A confident explanation must not silently replace the evidence from which it was generated.

Then define the lesson's scope. An instruction that helps one model complete a spreadsheet task should not automatically become policy for every assistant. Name the task, relevant tools and conditions under which the change applies. For changes affecting customer commitments, access or financial actions, retain approval with the responsible operating owner.

Test the old and proposed instructions on comparable cases. Include ordinary successful work, a temporary failure that should not change the preferred method, and a genuine change that should. Measure useful completion, avoidable errors, unnecessary escalation and review time. These are proposed acceptance checks, not a certified method established by either paper.

Keep changes individually reversible. If several lessons are installed together, a later improvement or regression may be difficult to explain. Record which lesson was admitted, the evidence used and the person or authorised process that accepted it. Review whether its benefit persists beyond the case that produced it.

Neither preprint provides a regional business-outcome comparison. The mechanism is relevant wherever an agent's written experience can influence later work, but the operating conditions must be established locally. A result from research tasks is not a performance estimate for North America, Europe, the United Kingdom or Asia-Pacific.

For multilingual operations, test whether a lesson preserves the meaning of instructions in the language actually used. Where connectivity or service availability is variable, including particular deployments in Africa, the Middle East and Latin America, distinguish a temporary interruption from a persistent change before teaching the assistant a new default. These are deployment-specific questions, not regional stereotypes or measured regional findings.

Local privacy, retention and sector requirements also constrain which task records may be kept or reused. Review those obligations separately; this briefing establishes no new legal rule. Start with one low-risk workflow. Before accepting its next AI-written lesson, ask: what happened, what does it justify changing, and how will we know the change helped?