SOPO: Socratic Guided Policy Optimization for Span-Level Hallucination Detection

Executive Summary & Core Discovery

SOPO introduces a multi-turn reinforcement learning framework designed to tackle the challenges of span-level hallucination detection in large language models (LLMs). The framework addresses localization ambiguity, reward sparsity on hard examples, and uncontrolled length inflation through a Socratic feedback mechanism and Turn-based Advantage Scaling. The Socratic feedback mechanism uses a teacher model to provide indirect hints inferred from the ground truth, enabling the model to generate more effective reasoning trajectories for hard examples. Turn-based Advantage Scaling penalizes verbosity bias, encouraging the model to develop efficient and autonomous reasoning capabilities. Empirical results on RAGTruth demonstrate that SOPO (4B) achieves state-of-the-art performance, surpassing leading proprietary models (e.g., GPT-5) and standard RL baselines, while obtaining higher reported mean scores than GRPO on four out-of-domain benchmarks, with mixed evidence against CAPO. The study releases the training and evaluation code alongside the 1.7B and 4B SOPO models.

Architectural Background & Theoretical Foundations

The SOPO framework builds upon the Reinforcement Learning with Verifiable Rewards (RLVR) paradigm, which has shown promise in reasoning tasks. However, applying RLVR to span-level hallucination detection presents unique challenges, including localization ambiguity, reward sparsity on hard examples, and uncontrolled length inflation. To address these issues, SOPO introduces a Socratic feedback mechanism and Turn-based Advantage Scaling. The Socratic feedback mechanism involves a teacher model that provides indirect hints inferred from the ground truth, guiding the reasoning trajectories during rollout. This approach helps alleviate reward sparsity by enabling the model to generate more effective reasoning trajectories for hard examples. Turn-based Advantage Scaling penalizes verbosity bias, encouraging the model to develop efficient and autonomous reasoning capabilities. By incorporating these mechanisms, SOPO aims to improve the robustness and effectiveness of span-level hallucination detection in LLMs.

Benchmark Architecture & Evaluation Methodology

The evaluation of SOPO is conducted on the RAGTruth benchmark, which is designed to assess the factuality and consistency of generated content. The benchmark includes a variety of tasks and datasets that challenge the model’s ability to detect and correct hallucinations. The evaluation methodology involves comparing SOPO against leading proprietary models and standard RL baselines. The performance metrics include reported mean scores and out-of-domain benchmark results. The study also releases the training and evaluation code alongside the 1.7B and 4B SOPO models, allowing for further research and validation.

Empirical Findings & Granular Performance Analysis

The empirical results demonstrate that SOPO achieves state-of-the-art performance in span-level hallucination detection. SOPO (4B) surpasses leading proprietary models, such as GPT-5, and standard RL baselines. The model obtains higher reported mean scores than GRPO on four out-of-domain benchmarks, with mixed evidence against CAPO. These results highlight the effectiveness of the Socratic feedback mechanism and Turn-based Advantage Scaling in improving the robustness and efficiency of hallucination detection. The study also provides a detailed analysis of the model’s performance across different tasks and datasets, offering insights into its strengths and limitations.

In-Depth Analysis of Core Limitations & Vulnerabilities

1. Localization Ambiguity

Localization ambiguity refers to the challenge of pinpointing the exact span of text that contains a hallucination. This ambiguity can lead to incorrect detection and correction of hallucinations, affecting the overall performance of the model. The Socratic feedback mechanism in SOPO helps mitigate this issue by providing indirect hints inferred from the ground truth, guiding the reasoning trajectories during rollout.

2. Reward Sparsity

Reward sparsity on hard examples is a significant challenge in reinforcement learning, particularly in tasks that require precise and nuanced reasoning. SOPO addresses this issue through its Socratic feedback mechanism, which enables the model to generate more effective reasoning trajectories for hard examples. This approach helps alleviate reward sparsity by providing indirect hints inferred from the ground truth.

3. Uncontrolled Length Inflation

Uncontrolled length inflation refers to the tendency of models to generate excessively long outputs, which can be inefficient and may not be necessary for the task at hand. Turn-based Advantage Scaling in SOPO penalizes verbosity bias, encouraging the model to develop efficient and autonomous reasoning capabilities. This mechanism helps mitigate uncontrolled length inflation by promoting concise and relevant outputs.

4. Verbosity Bias

Verbosity bias refers to the tendency of models to produce verbose and unnecessary outputs, which can be inefficient and may not be necessary for the task at hand. Turn-based Advantage Scaling in SOPO penalizes verbosity bias, encouraging the model to develop efficient and autonomous reasoning capabilities. This mechanism helps mitigate verbosity bias by promoting concise and relevant outputs.

Enterprise Production & Red-Teaming Blueprint

For enterprise deployment, the SOPO framework can be integrated into existing LLM evaluation pipelines to enhance the detection and correction of span-level hallucinations. The framework’s Socratic feedback mechanism and Turn-based Advantage Scaling can be adapted to various tasks and datasets, making it a versatile tool for improving the robustness and efficiency of hallucination detection. Regular evaluation and validation using the RAGTruth benchmark can ensure the model’s performance and reliability. Additionally, the training and evaluation code released by the study can be used to further validate and improve the model’s performance.

Technical FAQ (Granular & Non-Repetitive)

Takeaway

SOPO sets a new standard for span-level hallucination detection in LLMs, offering a robust framework that addresses critical challenges in the field. The Socratic feedback mechanism and Turn-based Advantage Scaling provide a versatile tool for improving the robustness and efficiency of hallucination detection, making it a valuable addition to enterprise LLM evaluation pipelines.