If the answer depends on your latest documents, start by helping the system find the right source. If it keeps answering in the wrong way, investigate its behaviour. Retrieval and fine-tuning solve different parts of the problem.
Evaluate current answers against business questions
Take an answer your team did not trust and work backwards. Was the source missing? Was it out of date? Did the system find the right paragraph but explain it badly? Each of these points to a different fix.
This review does not need to begin with a large platform. A handful of real questions, their approved answers and the source documents can tell us a great deal about the product people actually need.
Retrieve facts from authorised company sources
Retrieval-augmented generation—usually shortened to RAG—brings selected source passages into the model’s context. OpenAI’s retrieval documentation describes searching indexed files for relevant content.
The useful work around it includes keeping documents current, splitting them sensibly, finding the right passages and respecting access permissions. Showing a source link helps users investigate an answer, but the link alone does not prove that every sentence is correct.
Use fine-tuning to adjust model behaviour
Fine-tuning adapts a model using examples. It can be useful for a consistent response style, classification scheme or specialised task. OpenAI’s optimisation guidance treats evaluation and prompting as part of the decision too.
A changed handbook needs an updated source. A recurring formatting problem may need a better prompt, a validator or training examples. Separating those questions prevents an expensive technique from being used to solve the wrong problem.
Provide sources and escalation options
Test whether the system finds the source, respects access and gives a useful answer. Include questions it cannot answer and cases where documents disagree. An honest “I cannot establish that from these sources” is often a better product response than a plausible guess.
Keep the review set as the product evolves. The opportunity is not simply a chat box over files; it may be a faster policy check, a clearer research workflow or a new way for staff to navigate expertise.
Before you start
- Separate missing knowledge from inconsistent behaviour.
- Define authoritative sources, access and update ownership.
- Test retrieval quality and answer quality separately.
- Include unsupported questions and conflicting documents.
Related engineering work
Our Infinite Growth practice turns website and search findings into prioritised reports. It is a relevant example of making source information actionable, not evidence of a particular RAG or fine-tuning architecture.
View the project case