This AI eval metrics glossary defines the terms evaluation teams actually use: faithfulness, answer relevance, context precision, groundedness, and more. One plain sentence each, no academic fog.
We grade outputs. We do not train models, run fine-tuning, or sell experts we cannot verify.
An AI eval metric is a named dimension along which you judge a model output: one thing you measure, like whether the answer stuck to the facts or answered the question asked. Metrics turn "is this good?" into specific, gradeable questions. This glossary covers the ones you will meet in LLM evaluation work, each in one plain sentence.
Metrics are tools. Used wrong, they measure the wrong thing precisely.
One score to rule the product. A single number always hides the dimension that matters most to you, because no one metric captures correctness, usefulness, and safety at once. Measure several; report the distribution.
The dashboard says "quality: 87." Quality of what, measured how, against which standard? A metric without a definition is a rumor with a decimal point. Every metric you track should have a one-sentence definition, like the ones below.
A chatbot team tracking code-generation benchmarks, or a RAG team ignoring grounding because the generic eval suite did not include it. Metrics should match your product's failure modes, not someone else's leaderboard.
Running an LLM judge on every metric and never spot-checking it against human grades. Automated metrics drift, misunderstand rubrics, and reward their own blind spots. Validate them against human review regularly.
If your system retrieves documents and then answers from them, these are the metrics that matter most. They all ask one question in different ways: did the answer stay honest to the sources?
Whether every claim in the answer is supported by the retrieved context, with nothing invented or smuggled in from outside the provided sources.
Practically the same idea as faithfulness: the degree to which the answer is anchored in provided sources rather than made up. Teams use the two terms interchangeably.
Whether the retrieved documents that actually mattered were ranked above the irrelevant ones, so the answer was built on the right material.
Whether the retrieval step found all the documents needed to answer the question, as opposed to missing the one chunk that held the key fact.
Whether the sources the answer cites really say what the answer claims they say, checked citation by citation.
These measure the answer itself: did it address the question, say it well, and say neither too much nor too little?
Whether the answer actually addresses the question asked, rather than answering a nearby easier question or wandering off topic.
Whether the output obeyed the format, length, and structural constraints it was given, like returning valid JSON or staying under a word limit.
Whether the answer covers everything the question needed, with no important part dropped or hand-waved.
Whether the answer says what it needs to without padding, repetition, or filler that wastes the reader's time.
Whether the text reads naturally: grammatical, well-formed sentences that a person would actually write.
Whether the answer holds together logically from start to finish, with each part connecting to the next instead of contradicting it.
These measure what the output must never do. They are usually graded as pass or fail rather than on a sliding scale, because one failure outweighs many successes.
Whether the output contains hateful, harassing, or abusive language toward any person or group.
Whether the output could enable real-world harm: instructions for wrongdoing, dangerous advice, or content unsafe for its audience.
Whether the output treats people or groups unfairly, leaning on stereotypes instead of the facts of the individual case.
Whether the model refused when it should have refused, and answered when it should have answered, instead of refusing harmless requests or complying with harmful ones.
You send 20 to 50 model outputs plus your rubric, or we help you write one. Tell us which metrics matter to you.
Trained reviewers grade every sample against the rubric, backed by automated checks.
Sample-level grades, issue patterns, recommended fixes, and a walkthrough call.
It depends on your product. RAG systems live and die by faithfulness and context precision. Customer-facing chatbots need answer relevance and safety metrics. Code assistants need correctness above all. Start from your failure modes, not from a generic list, and pick the metrics that would have caught your last incident.
Many can, using an LLM as a judge or with purpose-built scorers, and automation is the right call at scale. But automated scores need validation against human grades before you trust them, and re-validation whenever your product changes. Our automated evaluation page covers how to do this honestly.
Faithfulness asks whether the answer is true to its sources. Answer relevance asks whether it answers the question. An answer can be faithful but irrelevant, quoting sources correctly while ignoring what was asked, or relevant but unfaithful, nailing the question with invented facts. You need both.
Pick your metrics, then write each one out with severity levels and examples, exactly like the worked example in our rubric writing guide. A metric is a dimension; a rubric is the dimension plus the grading instructions that make it repeatable.
Yes. Tell us which metrics matter for your product when you start a pilot, and we build them into the rubric. The report then shows per-sample grades on each metric, so you can see exactly where the model is strong and where it is not.
Be wary of any metric you cannot define in one sentence, and of composite scores that blend unrelated dimensions into a single number. If you cannot explain what a metric means to a new hire in thirty seconds, it will not survive contact with real grading.
Send 20 to 50 samples. Get every one graded on the metrics your product needs.
Not sure where to start? Talk to us and we will point you at the right evaluation.