The AI Assistant Was the Easy Part

Connecting a model to files did not solve the problem. Lenovo says the system needed to retrieve from multiple sources, remain reliable as knowledge changed and make the complexity invisible to employees. The durable work was deciding which sources counted, how they stayed current and how the service could grow without being rebuilt for every use case. ¹²

That turns enterprise knowledge from a collection of documents into an operating service. Every included source needs an authority, a review rhythm and a retirement path. If those decisions are absent, the assistant may make stale or contradictory material easier to consume.

Lenovo describes role-based and source-level permissions, plus supporting citations on more than 85% of answers. A citation helps an employee understand where an answer came from. It does not prove that the source is current or that the answer accurately represents it. Permission parity and source checking remain separate controls. ²

Require the assistant to inherit the effective access rights of the source and the user. Then measure citation correctness, not citation presence alone: does the cited material actually support the answer, and can the user reach it? A plausible response with an inaccessible or irrelevant citation is not trusted knowledge.

The case describes a feedback loop through which users flag weak responses and issues are routed to content, product or technical owners. That is more consequential than a generic thumbs-down button. A content problem requires an authoritative source decision. A product problem may require a better workflow. A technical problem may require retrieval or model changes. ¹²

Give each route an owner and response target. Track recurrent topics, stale sources, permission failures, unsupported answers and unresolved questions. The correction queue is not a support afterthought; it is how the knowledge service learns what the organisation does not yet know clearly enough.

Lenovo involved employees and domain experts in use-case design, then supported the service with training and change management. It reports 60-fold growth to approximately 3,000 users. The case does not disclose the active-use threshold or adoption period, so the multiplier should not be read as a general benchmark. ¹²

A new arXiv preprint adds a useful caution. In one large firm, researchers analysed 713,564 prompts and responses from nearly 4,000 back-office employees across 15 functions. They found no improvement in sophisticated use over time or lasting improvement after formal training. ⁸ It is one firm and a version-one preprint, but it supports a practical rule: learning must be embedded in the job, reinforced by experts and connected to a useful service.

Lenovo reports 30% less time spent on knowledge retrieval, 35% fewer escalations to subject-matter experts and twice-as-fast sales proposal preparation. It estimates 120 hours returned per employee annually and up to US$17 million in potential value. ¹²

The public material does not disclose the baseline, comparison group, study period, confidence intervals or the organisation and method behind the value validation. The financial figure is potential value, not audited realised savings. Leaders should therefore reproduce the measurement locally: define the retrieval task, measure time and escalation before launch, hold quality constant and show where any released capacity went.

A current Salesforce survey of 2,025 agentic-AI decision-makers across 20 countries likewise associates relevant data, bounded scope and pre-defined escalation with better self-reported outcomes. ⁶ The survey is vendor-sponsored and its outcomes are self-reported, so it supports testing those operating conditions—not a causal or universal return claim.

The United Kingdom's Ministry of Justice offers a useful comparison. Its cited knowledge-assistant pilot searches more than 300 operational documents, while semantic search is being scaled for probation and justice staff. GOV.UK Chat's two public pilots separately measured accuracy, usefulness, latency, safety and user hand-off across more than 10,000 users and 26,000 questions. ³⁴

Those public-service cases carry different consequences from an enterprise sales workflow, but the operating pattern travels: bounded sources, citations, human decisions and visible evaluation. What changes by market is the difficulty of doing it. Multilingual records, fragmented identity, uneven source quality, records law and limited subject-matter capacity can make ownership more expensive than the interface.

CorporateBench, a new version-one arXiv preprint, evaluates five models across four synthetic firms ranging from 12 to 10,000 employees and document collections exceeding 230,000 items. Performance worsened as the collection approached realistic scale. ⁷ The firms are synthetic and the paper is not peer reviewed, so it is not production proof.

The direction is still useful for procurement. Test the exact document volume, time changes, languages, access rules and conflicting sources that the service will face. A demonstration over a clean sample cannot establish enterprise reliability. NIST's voluntary generative-AI profile likewise recommends reassessment when retrieval augmentation or a third-party model changes the deployed system. ⁵

Choose one question set that matters to a real team. Name the five owners. Inventory the authoritative sources and conflicting versions. Test permission parity with users who should and should not see sensitive material. Ask domain experts to score answer and citation correctness. Route every weak answer through the correction process and measure the time to resolution.

Only then compare models or vendors. Over 30 days, track repeated use, time to a source-backed answer, escalation volume, unsupported-answer rate, correction time and one business outcome. Scale the service when those measures improve together—not when the interface merely looks convincing.