Retrieval over a firm’s own documents is one of the few uses of a language model that pays for itself quickly. Somebody asks a question in plain language and gets back the passage that answers it, with the document and paragraph attached. The work it replaces is real, measurable and disliked.
The risk is real too, and almost all of it is decided by configuration settled before anything is indexed. These are the questions worth answering first.
1. Where does the data physically sit?
Indexing a document set means copying it. Know which service holds the copy, which country it is in, and what your obligations are about that. For a firm holding client matter data, medical records or member records, this is the first question and sometimes the last one.
2. Is your content excluded from training?
Not “we believe so”. Find the setting, and find the contract term. Both should exist and both should say the same thing. If a provider cannot give you a clear written answer, that is itself the answer.
3. Who can ask what?
A retrieval system will happily surface a document to somebody who was never meant to see it, because it does not know that. Permissions have to be carried through from the source into the index, or the index has to be scoped to a set where everyone is entitled to everything. Deciding this after go-live means re-indexing.
4. Can the answer be checked?
An answer without a source is a claim. Retrieval should return the passage it drew on, so the person reading can verify it in the time it takes to glance. That single design decision turns the tool from something that must be trusted into something that can be checked, which is the difference between a system a professional will use and one they will quietly abandon.
5. What is it replacing, and how will you know if it worked?
Write down the task, and how long it currently takes, before you build. If the task cannot be named, the project is a demonstration rather than a deliverable. If the time cannot be estimated, you will have no way to tell afterwards whether the thing was worth doing.
The rule underneath all five
Where a task is better served by a deterministic rule than by a model, write the rule. A model is the right tool for open-ended questions over unstructured documents. It is a poor and expensive substitute for a lookup.