How to know if a private AI system is useful

A private AI system can look impressive in a meeting and still fail the firm that bought it. The questions in a demonstration are chosen to work. The questions in a working week are not. Evaluation closes that gap. It asks whether the system helps with the firm’s real work, where it fails, and whether a change made it better or worse.

The method below draws on public research and guidance available in March 2026, and the diagram at the top of this article sets out its steps. It is a procedure, not a score. It produces evidence that a partner, not only an engineer, can read.

Useful for what, and for whom

Usefulness has no general measure. NIST’s AI Risk Management Framework starts with context: who uses the system, for what purpose, and with what consequences when it is wrong. Its generative AI profile, NIST AI 600-1, asks organisations to document the intended context of use with domain experts, and names confabulation as a core risk: fluent output that is false.

For a small firm, context fits on one page. List the tasks the system should support: finding a clause across past agreements, summarising correspondence before a meeting, drafting a first reply. For each task, name the person who can judge a good answer. Everything that follows is measured against those tasks, not against a public benchmark.

Build the question set from real work

The core of an evaluation is a fixed set of questions, drawn from real requests with the firm’s permission, and stripped of anything that need not be there. Each question records the documents that hold the answer and what a correct answer contains. Four kinds belong in the set:

1. Answerable questions. The answer sits in one or more documents the asker may read.
2. Unanswerable questions. Nothing in the collection answers them. The correct response says so. A system that invents an answer here fails the most important test.
3. Restricted questions. The answer exists, but in documents the asker may not see. The correct response reveals nothing from them.
4. Hard cases. Answers spread across several documents, superseded versions, tables, scans, and questions in a second language.

The set does not need to be large to be useful. It needs to cover the work. It stays fixed so results can be compared over time, with new cases added when real failures appear.

Check retrieval and answer separately

A retrieval system has two stages, and each fails in its own way. The Ragas paper separates the dimensions: whether retrieval finds relevant, focused passages; whether the answer stays faithful to those passages; and whether it addresses the question. Ragas proposes automated metrics for these. Automated scores help with volume, but on a small firm’s question set, expert reading is often both feasible and more trustworthy.

Separating the stages tells you where to act. If the right passage never arrives, the fix lies in the documents, the index or the permissions. If the right passage arrives and the answer departs from it, the fix lies in the model or the prompt.

A citation is a claim to check

Citations make an answer checkable. They do not make it true. Liu, Zhang and Liang audited four generative search engines in 2023 and found that, on average, only 51.5% of generated sentences were fully supported by their citations, and 74.5% of citations supported the sentence they were attached to. The ALCE benchmark, also from 2023, reported that on its ELI5 dataset even the best models of the time lacked complete citation support half of the time. These figures describe the systems and datasets those papers studied. They say nothing about any private system a firm might deploy now, which has to be measured on its own.

The lesson carries over. For each answer in the evaluation, the reviewer opens the cited passage and asks one question: does this passage support this sentence? A citation to a real document that does not say what the answer claims is harder to spot than no citation at all.

Let the experts judge, then decide

The people who do the work judge the answers. For each question, the reviewer records a plain verdict, with a sentence of reason:

• Correct and supported.
• Correct but unsupported, or supported by the wrong passage.
• Incomplete.
• Wrong, or invented.
• Correctly declined, or wrongly declined.
• Leaked restricted content.

The last category is a stop condition on its own. One leak through the permission model outweighs any number of good answers.

The results then meet a decision, made in advance, not after the fact: adopt for the tasks that passed, adjust and re-test, or stop. Some tasks will pass and others will not. A system approved for finding documents need not be approved for drafting replies.

Evaluation also sets how people should use the system. The OWASP LLM09:2025 Misinformation entry warns against overreliance and recommends cross-checking and human oversight for sensitive information. An interface that shows sources beside each answer, and training that expects staff to open them, puts that advice into daily practice.

Every change is a new system

A new model version, a new embedding model, a re-indexed collection or a new permission rule can each change answers. The question set runs again after every such change, and results are compared with the last run before release. The record of each run, with date, versions and verdicts, becomes the system’s history. When someone asks why the system behaves as it does, that history answers.

Live use adds evidence the question set cannot. A simple way to flag a bad answer, reviewed by a named person, feeds new cases into the set.

What this does not tell you

A passed evaluation shows how the system performed on these questions, with these documents and versions, on the day of the test. It does not show that every future answer will be correct. Answers still need the judgement of the person who uses them.

To design an evaluation around your firm’s work, write to contact@desgarde.com.