48 plain-English definitions across evaluation, data, safety, methods, and metrics. No vendor marketing, no hedging.
48 terms, 5 disciplines. Definitions are reviewed against current usage. If a term is missing, tell us and we will add it.
A · B · C · D · E · F · G · H · I · J · K · L · M · N · O · P · Q · R · S · T · U · V · W · Z
The discipline of making an AI system's behavior match human values, intent, and safety requirements. Alignment is not a checkpoint but a continuous process of evaluation, feedback, and correction, the reason human judgment sits inside every serious training loop.
The act of labeling data, grading outputs, tagging errors, ranking responses, that models learn from or are measured against. Annotation quality is the ceiling on model quality: expert annotation raises it; crowd annotation encodes its own errors into the model.
Field note. Field note When a client's model mysteriously plateaus, annotation quality is the first place we audit, and the usual culprit.
An input deliberately crafted to make a model fail, producing wrong answers, unsafe content, or leaked instructions, while looking entirely ordinary to automated defenses.
Test data leaking into training data, inflating benchmark scores that collapse in production. The quiet epidemic of modern LLM evaluation, and the reason held-out expert evaluation still matters.
Field note. Field note We treat any score that jumps more than two standard deviations on a public benchmark as contaminated until proven otherwise.
A paid program rewarding researchers or evaluators for discovering and responsibly disclosing model failures such as jailbreaks, hallucination patterns, or data leakage.
Two related meanings. For models: the alignment between expressed confidence and actual correctness. For raters: the process of aligning evaluators to a shared rubric, which is why calibrated cohorts beat large crowds.
Grading a model's intermediate reasoning steps rather than only its final answer, exposing sound-looking conclusions built on broken logic. Where domain experts earn their fee.
A chance-corrected measure of inter-rater agreement. Below roughly 0.8, labeled data injects more noise than signal into training and evaluation pipelines.
Field note. Field note Vendors who quote raw percent agreement instead of kappa are usually hiding disagreement. Ask for kappa.
A technique that steers models using a written set of principles plus AI feedback, reducing human labeling volume. Strong on policy adherence; still weaker than expert humans on specialist edge cases, per our 2026 comparison data.
A fine-tuning method that learns directly from human preference pairs without explicitly training a separate reward model, simpler, cheaper, and still entirely dependent on preference-data quality.
Transferring capability from a large teacher model to a smaller student model. Students inherit the teacher's blind spots unless independently verified by expert evaluation.
A pairwise-comparison rating system borrowed from chess, used by arena-style leaderboards. Its validity is bounded by the expertise of the voting pool, crowd votes measure crowd preferences.
The combined automated and human pipeline that runs a model against test suites, rubrics, and expert raters to produce a defensible picture of capability and safety.
A hallucinated reference with realistic metadata, plausible authors, journals, even DOIs, that never existed. The leading liability risk in legal and medical AI, and largely invisible to automated checkers.
Field note. Field note In our audits, fabricated citations with realistic metadata outnumber obviously fake ones roughly three to one. Plausibility is the threat.
Continued training of a pretrained model on curated data to specialize its behavior for a domain, output format, or safety policy.
The authoritative answer key against which outputs are judged. In specialist domains like medicine, law, and finance, producing it takes real domain knowledge, not just crowd labels.
Runtime constraints, filters, classifiers, policy checks, that block unsafe output before it reaches users. A complement to alignment, never a substitute for it.
Confident model output untethered from facts. Working taxonomies split it into fabricated (invented), extrapolated (overgeneralized), and inverted (directly contradicted) claims, each requiring different detection strategies.
A system design in which human judgment gates, corrects, or approves model decisions before they take effect, from RLHF rating loops to clinical decision support.
A measure of consistency between evaluators. For RLHF data, agreement below 90% is noise: the reward model learns disagreement, not judgment.
Field note. Field note IRA is the single best predictor of preference-data value, better than rater count, better than turnaround speed.
Supervised fine-tuning on instruction-response pairs so a base model becomes an assistant that follows directions rather than one that merely continues text.
A model adapting behavior from examples inside the prompt, with no weight updates. Powerful, but unverified and unstable as a capability claim until tested across prompt variations.
A prompt-level attack that bypasses safety training to extract content the model is supposed to refuse, from roleplay framing to cross-modal injection. Found at scale only by adversarial humans, not automated attacks.
See Cohen's kappa. The chance-corrected agreement statistic that separates real evaluator alignment from coincidence. The number serious vendors quote.
The date horizon of a model's training data, the root cause of stale-answer failures long after deployment, and the reason retrieval and freshness auditing exist.
A transformer-based model trained on text at massive scale, capable of generating fluent language. Fluency is orthogonal to accuracy, the central trap of modern AI products.
Errors in training or evaluation labels. Noise compounds silently: every downstream model inherits it and amplifies it, which is why cheap labeling is expensive.
A standardized document describing a model's capabilities, limitations, evaluation results, and intended use, the nutrition label of AI. Its honesty is only as good as the evaluation behind it.
Assessing models across text, image, audio, and mixed inputs. Failure modes multiply per modality, and so must the expertise of the evaluators judging them.
Arithmetic, unit, and quantity errors that read fluently, transposed digits, blended quarters, invented guidance figures. The failure mode that quietly poisons financial AI.
Optimizing a model against a fixed test until scores rise while real-world capability does not. Teaching to the test, at industrial scale.
Ranked comparisons of model responses, which answer is better, and why, used to train reward models and align behavior. The most valuable and most frequently corrupted data type in modern AI.
An adversarial payload hidden in input or retrieved context that hijacks model behavior. The defining security threat of the agent era, and the target of our fastest-growing red-team practice.
Expert narrative assessment of outputs, capturing nuance, tone, cultural fit, and domain soundness that no current metric encodes. Slow, expensive, and still unmatched for the failures that matter most.
Reinforcement Learning from Human Feedback: train a reward model on human preferences, then optimize the policy against it. The quality ceiling of RLHF is the expertise of its raters, the entire thesis of JudgeMyAI.
Field note. Field note RLHF doesn't fail at the algorithm; it fails at the rater. Fix the humans, and the math works.
Using AI feedback in place of (or alongside) human raters. It scales cheaply, and inherits every blind spot of the feedback model, which is precisely where expert auditing pays for itself.
Structured adversarial testing of a model by humans who attack it the way real adversaries would, surfacing jailbreaks, leakage, and emergent failure modes before deployment, not after the incident.
A model gaming its reward signal instead of fulfilling its intent, sycophancy, verbosity, and confident evasion are classic symptoms. Detection requires judges the model can't fool.
A learned proxy for human judgment that scores outputs during RLHF. Proxy quality bounds alignment quality, garbage preferences in, garbage proxy out.
Supervised training on curated demonstration data, typically the first step in turning a base model into an aligned assistant, before preference optimization.
A model telling users what they want to hear instead of what is true. Typically worsened by approval-style preference data, raters reward agreement, models learn to flatter.
Field note. Field note Our antidote: preference pairs where the agreeable answer is the wrong answer. Sycophancy dropped from 14.2% to 2.1% in FILE 02 of our case archive.
Methods that let weaker judges reliably supervise stronger systems, debate, task decomposition, and recursive verification among them. The frontier alignment problem, stated honestly.
Testing model behavior against harm taxonomies, usage policies, and regulatory standards before and after deployment, with documentation strong enough to hand to a regulator.
Measuring harmful output across categories such as hate, self-harm, and violence, where context determines whether a term is clinical, forensic, or dangerous, and only expert raters can draw the line.
Calibrated confidence and abstention: a model knowing, and admitting, what it does not know. The difference between a useful assistant and a confident liability.
Held-out data used to tune decisions during development. Leakage between train and validation destroys its meaning, and quietly, the credibility of every number derived from it.
The share of pairwise comparisons a model wins. Useful, directional, and exactly as trustworthy as the humans casting the votes.
Evaluating or operating a model on a task with no in-prompt examples, the honest test of transferred capability, and where untested claims go to die.
Once you know the terms, here is how to measure what actually matters.
An AI eval metric is a named dimension along which you judge a model output: one thing you measure, like whether the answer stuck to the facts or answered the question asked. Metrics turn "is this good?" into specific, gradeable questions. This glossary covers the ones you will meet in LLM evaluation work, each in one plain sentence.
Metrics are tools. Used wrong, they measure the wrong thing precisely.
One score to rule the product. A single number always hides the dimension that matters most to you, because no one metric captures correctness, usefulness, and safety at once. Measure several; report the distribution.
The dashboard says "quality: 87." Quality of what, measured how, against which standard? A metric without a definition is a rumor with a decimal point. Every metric you track should have a one-sentence definition, like the ones below.
A chatbot team tracking code-generation benchmarks, or a RAG team ignoring grounding because the generic eval suite did not include it. Metrics should match your product's failure modes, not someone else's leaderboard.
Running an LLM judge on every metric and never spot-checking it against human grades. Automated metrics drift, misunderstand rubrics, and reward their own blind spots. Validate them against human review regularly.
If your system retrieves documents and then answers from them, these are the metrics that matter most. They all ask one question in different ways: did the answer stay honest to the sources?
Whether every claim in the answer is supported by the retrieved context, with nothing invented or smuggled in from outside the provided sources.
Practically the same idea as faithfulness: the degree to which the answer is anchored in provided sources rather than made up. Teams use the two terms interchangeably.
Whether the retrieved documents that actually mattered were ranked above the irrelevant ones, so the answer was built on the right material.
Whether the retrieval step found all the documents needed to answer the question, as opposed to missing the one chunk that held the key fact.
Whether the sources the answer cites really say what the answer claims they say, checked citation by citation.
These measure the answer itself: did it address the question, say it well, and say neither too much nor too little?
Whether the answer actually addresses the question asked, rather than answering a nearby easier question or wandering off topic.
Whether the output obeyed the format, length, and structural constraints it was given, like returning valid JSON or staying under a word limit.
Whether the answer covers everything the question needed, with no important part dropped or hand-waved.
Whether the answer says what it needs to without padding, repetition, or filler that wastes the reader's time.
Whether the text reads naturally: grammatical, well-formed sentences that a person would actually write.
Whether the answer holds together logically from start to finish, with each part connecting to the next instead of contradicting it.
These measure what the output must never do. They are usually graded as pass or fail rather than on a sliding scale, because one failure outweighs many successes.
Whether the output contains hateful, harassing, or abusive language toward any person or group.
Whether the output could enable real-world harm: instructions for wrongdoing, dangerous advice, or content unsafe for its audience.
Whether the output treats people or groups unfairly, leaning on stereotypes instead of the facts of the individual case.
Whether the model refused when it should have refused, and answered when it should have answered, instead of refusing harmless requests or complying with harmful ones.
You send 20 to 50 model outputs plus your rubric, or we help you write one. Tell us which metrics matter to you.
Trained reviewers grade every sample against the rubric, backed by automated checks.
Sample-level grades, issue patterns, recommended fixes, and a walkthrough call.
It depends on your product. RAG systems live and die by faithfulness and context precision. Customer-facing chatbots need answer relevance and safety metrics. Code assistants need correctness above all. Start from your failure modes, not from a generic list, and pick the metrics that would have caught your last incident.
Many can, using an LLM as a judge or with purpose-built scorers, and automation is the right call at scale. But automated scores need validation against human grades before you trust them, and re-validation whenever your product changes. Our automated evaluation page covers how to do this honestly.
Faithfulness asks whether the answer is true to its sources. Answer relevance asks whether it answers the question. An answer can be faithful but irrelevant, quoting sources correctly while ignoring what was asked, or relevant but unfaithful, nailing the question with invented facts. You need both.
Pick your metrics, then write each one out with severity levels and examples, exactly like the worked example in our rubric writing guide. A metric is a dimension; a rubric is the dimension plus the grading instructions that make it repeatable.
Yes. Tell us which metrics matter for your product when you start a pilot, and we build them into the rubric. The report then shows per-sample grades on each metric, so you can see exactly where the model is strong and where it is not.
Be wary of any metric you cannot define in one sentence, and of composite scores that blend unrelated dimensions into a single number. If you cannot explain what a metric means to a new hire in thirty seconds, it will not survive contact with real grading.
Send 20 to 50 samples. Get every one graded on the metrics your product needs.
Not sure where to start? Talk to us and we will point you at the right evaluation.