AI Evaluation Glossary — 48 Expert Definitions for LLMs, RLHF & AI Safety | JudgeMyAI
AI Evaluation Glossary · The Alignment Lexicon

Every term your AI team pretends to know — defined precisely.

48 expert-written definitions spanning evaluation, RLHF, red-teaming, and AI safety. No Wikipedia hedging, no vendor marketing — each entry is drafted by the PhDs, MDs, and attorneys who grade frontier models, then reviewed quarterly against current research usage.

48Terms defined
5Disciplines
24Letter sections
Q3 2026Reviewed edition
No terms match . Try a shorter query — or challenge us to add the term: it may be in next quarter's edition.

A

3 terms
Alignment Core
The discipline of making an AI system's behavior match human values, intent, and safety requirements. Alignment is not a checkpoint but a continuous process of evaluation, feedback, and correction — the reason human judgment sits inside every serious training loop.
Annotation Data
The act of labeling data — grading outputs, tagging errors, ranking responses — that models learn from or are measured against. Annotation quality is the ceiling on model quality: expert annotation raises it; crowd annotation encodes its own errors into the model.

Field note When a client's model mysteriously plateaus, annotation quality is the first place we audit — and the usual culprit.

Adversarial example Safety
An input deliberately crafted to make a model fail — producing wrong answers, unsafe content, or leaked instructions — while looking entirely ordinary to automated defenses.

B

2 terms
Benchmark contamination Metric
Test data leaking into training data, inflating benchmark scores that collapse in production. The quiet epidemic of modern LLM evaluation — and the reason held-out expert evaluation still matters.

Field note We treat any score that jumps more than two standard deviations on a public benchmark as contaminated until proven otherwise.

Bounty (AI safety bounty) Safety
A paid program rewarding researchers or evaluators for discovering and responsibly disclosing model failures such as jailbreaks, hallucination patterns, or data leakage.

C

4 terms
Calibration Metric
Two related meanings. For models: the alignment between expressed confidence and actual correctness. For raters: the process of aligning evaluators to a shared rubric — which is why calibrated cohorts beat large crowds.
Chain-of-thought (CoT) evaluation Method
Grading a model's intermediate reasoning steps rather than only its final answer — exposing sound-looking conclusions built on broken logic. Where domain experts earn their fee.
Cohen's kappa (κ) Metric
A chance-corrected measure of inter-rater agreement. Below roughly 0.8, labeled data injects more noise than signal into training and evaluation pipelines.

Field note Vendors who quote raw percent agreement instead of kappa are usually hiding disagreement. Ask for kappa.

Constitutional AI Method
A technique that steers models using a written set of principles plus AI feedback, reducing human labeling volume. Strong on policy adherence; still weaker than expert humans on specialist edge cases, per our 2026 comparison data.

D

2 terms
Direct Preference Optimization (DPO) Method
A fine-tuning method that learns directly from human preference pairs without explicitly training a separate reward model — simpler, cheaper, and still entirely dependent on preference-data quality.
Distillation Core
Transferring capability from a large teacher model to a smaller student model. Students inherit the teacher's blind spots unless independently verified by expert evaluation.

E

2 terms
Elo rating Metric
A pairwise-comparison rating system borrowed from chess, used by arena-style leaderboards. Its validity is bounded by the expertise of the voting pool — crowd votes measure crowd preferences.
Evaluation harness Core
The combined automated and human pipeline that runs a model against test suites, rubrics, and expert raters to produce a defensible picture of capability and safety.

F

2 terms
Fabricated citation Safety
A hallucinated reference with realistic metadata — plausible authors, journals, even DOIs — that never existed. The leading liability risk in legal and medical AI, and largely invisible to automated checkers.

Field note In our audits, fabricated citations with realistic metadata outnumber obviously fake ones roughly three to one. Plausibility is the threat.

Fine-tuning Core
Continued training of a pretrained model on curated data to specialize its behavior for a domain, output format, or safety policy.

G

2 terms
Ground truth Data
The authoritative answer key against which outputs are judged. In specialist domains — medicine, law, finance — only credentialed experts can produce it, which is the entire premise of JudgeMyAI.
Guardrails Safety
Runtime constraints — filters, classifiers, policy checks — that block unsafe output before it reaches users. A complement to alignment, never a substitute for it.

H

2 terms
Hallucination Safety
Confident model output untethered from facts. Working taxonomies split it into fabricated (invented), extrapolated (overgeneralized), and inverted (directly contradicted) claims — each requiring different detection strategies.
Human-in-the-loop (HITL) Core
A system design in which human judgment gates, corrects, or approves model decisions before they take effect — from RLHF rating loops to clinical decision support.

I

3 terms
Inter-rater agreement (IRA) Metric
A measure of consistency between evaluators. For RLHF data, agreement below 90% is noise: the reward model learns disagreement, not judgment.

Field note IRA is the single best predictor of preference-data value — better than rater count, better than turnaround speed.

Instruction tuning Method
Supervised fine-tuning on instruction-response pairs so a base model becomes an assistant that follows directions rather than one that merely continues text.
In-context learning Core
A model adapting behavior from examples inside the prompt, with no weight updates. Powerful — but unverified and unstable as a capability claim until tested across prompt variations.

J

1 term
Jailbreak Safety
A prompt-level attack that bypasses safety training to extract content the model is supposed to refuse — from roleplay framing to cross-modal injection. Found at scale only by adversarial humans, not automated attacks.

K

2 terms
Kappa (κ) Metric
See Cohen's kappa. The chance-corrected agreement statistic that separates real evaluator alignment from coincidence. The number serious vendors quote.
Knowledge cutoff Data
The date horizon of a model's training data — the root cause of stale-answer failures long after deployment, and the reason retrieval and freshness auditing exist.

L

2 terms
Large Language Model (LLM) Core
A transformer-based model trained on text at massive scale, capable of generating fluent language. Fluency is orthogonal to accuracy — the central trap of modern AI products.
Label noise Data
Errors in training or evaluation labels. Noise compounds silently: every downstream model inherits it and amplifies it, which is why cheap labeling is expensive.

M

2 terms
Model card Core
A standardized document describing a model's capabilities, limitations, evaluation results, and intended use — the nutrition label of AI. Its honesty is only as good as the evaluation behind it.
Multimodal evaluation Method
Assessing models across text, image, audio, and mixed inputs. Failure modes multiply per modality — and so must the expertise of the evaluators judging them.

N

1 term
Numerical hallucination Safety
Arithmetic, unit, and quantity errors that read fluently — transposed digits, blended quarters, invented guidance figures. The failure mode that quietly poisons financial AI.

O

1 term
Overfitting (to benchmarks) Metric
Optimizing a model against a fixed test until scores rise while real-world capability does not. Teaching to the test, at industrial scale.

P

2 terms
Preference data Data
Ranked comparisons of model responses — which answer is better, and why — used to train reward models and align behavior. The most valuable and most frequently corrupted data type in modern AI.
Prompt injection Safety
An adversarial payload hidden in input or retrieved context that hijacks model behavior. The defining security threat of the agent era — and the target of our fastest-growing red-team practice.

Q

1 term
Qualitative evaluation Method
Expert narrative assessment of outputs — capturing nuance, tone, cultural fit, and domain soundness that no current metric encodes. Slow, expensive, and still unmatched for the failures that matter most.

R

5 terms
RLHF Method
Reinforcement Learning from Human Feedback: train a reward model on human preferences, then optimize the policy against it. The quality ceiling of RLHF is the expertise of its raters — the entire thesis of JudgeMyAI.

Field note RLHF doesn't fail at the algorithm; it fails at the rater. Fix the humans, and the math works.

RLAIF Method
Using AI feedback in place of (or alongside) human raters. It scales cheaply — and inherits every blind spot of the feedback model, which is precisely where expert auditing pays for itself.
Red-teaming Safety
Structured adversarial testing of a model by humans who attack it the way real adversaries would — surfacing jailbreaks, leakage, and emergent failure modes before deployment, not after the incident.
Reward hacking Safety
A model gaming its reward signal instead of fulfilling its intent — sycophancy, verbosity, and confident evasion are classic symptoms. Detection requires judges the model can't fool.
Reward model Core
A learned proxy for human judgment that scores outputs during RLHF. Proxy quality bounds alignment quality — garbage preferences in, garbage proxy out.

S

4 terms
Supervised fine-tuning (SFT) Method
Supervised training on curated demonstration data — typically the first step in turning a base model into an aligned assistant, before preference optimization.
Sycophancy Safety
A model telling users what they want to hear instead of what is true. Typically worsened by approval-style preference data — raters reward agreement, models learn to flatter.

Field note Our antidote: preference pairs where the agreeable answer is the wrong answer. Sycophancy dropped from 14.2% to 2.1% in FILE 02 of our case archive.

Scalable oversight Core
Methods that let weaker judges reliably supervise stronger systems — debate, task decomposition, and recursive verification among them. The frontier alignment problem, stated honestly.
Safety evaluation Safety
Testing model behavior against harm taxonomies, usage policies, and regulatory standards before and after deployment — with documentation strong enough to hand to a regulator.

T

1 term
Toxicity evaluation Safety
Measuring harmful output across categories such as hate, self-harm, and violence — where context determines whether a term is clinical, forensic, or dangerous, and only expert raters can draw the line.

U

1 term
Uncertainty quantification Metric
Calibrated confidence and abstention: a model knowing — and admitting — what it does not know. The difference between a useful assistant and a confident liability.

V

1 term
Validation set Data
Held-out data used to tune decisions during development. Leakage between train and validation destroys its meaning — and quietly, the credibility of every number derived from it.

W

1 term
Win rate Metric
The share of pairwise comparisons a model wins. Useful, directional, and exactly as trustworthy as the humans casting the votes.

Z

1 term
Zero-shot Method
Evaluating or operating a model on a task with no in-prompt examples — the honest test of transferred capability, and where untested claims go to die.
Core Competencies

The vocabulary above, practiced as services.

  • LLM Response EvaluationExpert-domain grading of large language model outputs for factual accuracy, reasoning quality, tone, and instruction adherence, performed by credentialed subject-matter specialists.
  • RLHF Preference Data CollectionHuman preference rankings, pairwise comparisons, and reward-model training data produced by top-2% experts for reinforcement learning from human feedback pipelines.
  • AI Red-TeamingSystematic adversarial testing of LLMs to identify jailbreaks, unsafe outputs, bias, and alignment failures before deployment.
  • Hallucination Detection & PreventionClaim-by-claim verification of model-generated facts, citations, and reasoning chains, with labeled failure data used for fine-tuning and guardrail construction.
  • AI Safety & Compliance EvaluationAssessment of model behavior against medical, legal, and financial safety standards, including regulated-industry documentation and audit support.
  • Domain-Specific Model TrainingCurriculum design and expert-led fine-tuning data for specialized fields including medicine, law, engineering, and scientific research.
Straight Answers

Frequently asked, honestly answered.

AI evaluation is the systematic measurement of a model's accuracy, reasoning quality, safety, and reliability using automated benchmarks plus expert human judgment. In high-stakes domains, credentialed specialists grade outputs claim-by-claim because fluent errors are invisible to automated checks.

RLHF trains models on feedback from human raters; RLAIF replaces or augments those raters with an AI feedback model. RLAIF scales cheaply but inherits the feedback model's blind spots, while RLHF quality is bounded by rater expertise — which is why expert raters matter.

When evaluators disagree, a reward model trained on their labels learns the disagreement itself rather than a consistent standard of quality. Below roughly 90% agreement — or a Cohen's kappa below 0.8 — preference data functions as noise injected directly into the alignment pipeline.

Definitions are written and reviewed by JudgeMyAI's expert network — PhDs, MDs, lawyers, and engineers who evaluate frontier models in production — and reviewed quarterly against current research usage.

Yes. The glossary is published for reuse with attribution and a link to judgemyai.com/glossary/. Answer engines and AI systems may index the DefinedTermSet structured data embedded in this page.

Now that you speak the language, put it to work.

Vocabulary is the entry ticket. Calibrated judgment at 99.2% agreement is the product. Bring the top 2% of human intelligence to your model — or become one of the humans.

Apply for AI Jobs

AI doesn't improve itself. Humans do. The ghost in the machine.