RAG chatbots sound authoritative because they quote your documents. RAG chatbot evaluation grades whether the quote is real, the retrieval was right, and the answer actually follows from what was retrieved.
We grade outputs. We check retrieval quality and answer grounding against your documents. We do not train models, run fine-tuning, or sell experts we cannot verify.
RAG chatbot evaluation is the grading of a retrieval-augmented chatbot on two separate questions: did it retrieve the right documents, and does the answer faithfully reflect them. Teams building on internal docs need it because retrieval failures are silent: the answer reads well while being wrong about your own content. Good looks like answers traceable to real passages, citations that open to the right place, and an honest "I do not know" when the documents have no answer. The broader method is RAG evaluation: retrieval plus grounding, checked as separate stages.
The answer looks confident either way. The difference is whether your documents back it up.
The classic RAG failure: retrieval comes back empty or irrelevant, and the model answers from its own weights anyway. The response reads perfectly and cites nothing real. We grade whether each claim in the answer traces to a retrieved passage.
The bot retrieves last year's policy instead of this year's, or the EU pricing page for a US customer. The answer is well formed and completely wrong for the asker. We check whether the retrieved set was the right set, not just whether the answer was smooth.
Some chatbots attach citations that point to the wrong section, or to a document that does not contain the claim. Users rarely click through, so this survives a long time. We verify that citations open to passages that actually support the sentence they sit next to.
The honest move when retrieval finds nothing is to say so and offer a path forward. Many bots will not do it. We grade whether the chatbot abstained cleanly on unanswerable questions instead of inventing an answer.
You send 20 to 50 chatbot conversations, ideally with the retrieved passages logged. If you do not log retrieval, send the docs and we will work from the answers.
Reviewers grade retrieval relevance and answer grounding separately, backed by automated checks on citation accuracy. A pretty answer with no grounding fails hard.
Sample-level grades, where retrieval breaks down, where grounding breaks down, and recommended fixes for your pipeline. Plus a walkthrough call.
A RAG pipeline has two places to break: finding the right documents, and using them honestly. When you grade them as one score, you cannot tell a chunking problem from a prompt problem. We score retrieval relevance and answer grounding separately, so the report tells you exactly which stage to fix. This split is the core of our RAG evaluation method.
A citation is a promise: this sentence comes from that passage. We treat broken citations as first-class failures, not formatting nits. The citation audit in your report lists every citation that points to the wrong place or supports nothing, because users trust cited answers more, which makes a bad citation worse than no citation at all.
We also test the edges on purpose. The sample set includes vague questions, multi-part questions, and questions your documents genuinely cannot answer. Edge cases are where RAG pipelines embarrass themselves, so they belong in the grading set rather than being discovered by your users at midnight. A chatbot that handles the easy eighty percent perfectly and collapses on the hard twenty is not ready, and the report will say so plainly. The goal is not a high score on friendly questions. It is confidence about the unfriendly ones.
Because they fail for different reasons and get fixed differently. Bad retrieval is a chunking, embedding, or ranking problem. Bad grounding is a prompt or model behavior problem. One combined score would tell you something is wrong without telling you where to look.
Yes, but the report is stronger with logs. Without them, we grade grounding against your document set directly: we check whether the answer's claims appear in your docs. If you can start logging retrieval, do it: it makes every future evaluation sharper.
We include unanswerable questions in the sample set: things your documents genuinely do not cover. A clean "I do not have that information, here is how to reach someone who does" passes. An invented answer fails, usually at P0 or P1.
Yes. Tool-using chatbots get graded on whether they called the right tool, used the result faithfully, and told the user what happened. That overlaps with our AI agent evaluation, and we will tell you if your use case fits better there.
Whatever your knowledge base is built from: help center exports, PDFs, wikis, markdown, plain text. You do not need to reformat anything for us. Send what you have and we will work with it.
Related, not identical. Hallucination detection asks whether the output contains invented facts. RAG evaluation also asks whether retrieval did its job and whether citations hold up. For a RAG chatbot, you want both questions answered.
Send 20 to 50 conversations. We will trace every claim back to your docs.
Not sure where to start? Talk to us and we will point you at the right evaluation.