Compaction Strategies in Long Horizon Agentic Tasks: A Comprehensive Analysis

Executive Summary & Core Discovery

The study by Jeanne Salle et al. reveals that compaction strategies in long horizon agentic tasks have minimal impact on task performance for large context models, but significant constraints can be silently lost. This finding underscores the need for robust evaluation frameworks to ensure the integrity of agentic tasks.

Architectural Background & Theoretical Foundations

The study focuses on long horizon agentic tasks, which are pushing LLM agents past their context window. Compaction is a critical process that involves freeing up the context window, but it is generally lossy by design. The study implements five compaction strategies within the OpenCode agent scaffold and evaluates three frontier models on three long horizon benchmarks: PaperBench, ProgramBench, and PostTrainBench.

Benchmark Architecture & Evaluation Methodology

The evaluation methodology involves testing three frontier models with varying context window sizes on three long horizon benchmarks. The study measures the impact of different compaction thresholds and strategies on task performance and constraint retention. The benchmarks include PaperBench, ProgramBench, and PostTrainBench, which are designed to evaluate the models’ ability to handle long horizon tasks.

Empirical Findings & Granular Performance Analysis

The empirical findings show that for the two 1M context models, the compaction threshold does not measurably affect task performance. However, GPT-5.3-Codex, the model with the smallest context window, performs worse at lower thresholds on PaperBench and, less clearly, on PostTrainBench. No compaction method outperforms default self-summarization, and methods that discard more content cost some models up to 20 points while leaving other models unchanged.

In-Depth Analysis of Core Limitations & Vulnerabilities

The study identifies four core limitations in the current compaction strategies for long horizon agentic tasks. These limitations include the lossy nature of compaction, the impact of compaction on constraint retention, the variability in performance across different models, and the need for robust evaluation frameworks to ensure the integrity of agentic tasks.

1. Lossy Nature of Compaction

Compaction is generally lossy by design, which can lead to the silent loss of task constraints, instructions, or reasoning behind earlier decisions. This limitation highlights the need for more robust compaction strategies that minimize information loss.

2. Impact on Constraint Retention

Compaction can silently remove task constraints, which is particularly problematic for long horizon agentic tasks. The study found that on ProgramBench, the first compaction summary already drops about one third of the task’s anticheating rules on average, and this rises monotonically with the number of compactions. This limitation underscores the need for compaction strategies that prioritize constraint retention.

3. Variability in Performance Across Models

The study found that the impact of compaction on task performance varies significantly across different models. For the two 1M context models, the compaction threshold does not measurably affect task performance, whereas GPT-5.3-Codex, the model with the smallest context window, performs worse at lower thresholds on PaperBench and, less clearly, on PostTrainBench. This limitation highlights the need for model-specific compaction strategies.

4. Need for Robust Evaluation Frameworks

The study highlights the need for robust evaluation frameworks to ensure the integrity of agentic tasks. The current benchmarks are compaction-friendly, meaning that the objective is fixed and the agent’s progress stays on disk, so most of what compaction discards can be recovered, except constraints that don’t exist anywhere else. This limitation underscores the need for more comprehensive evaluation frameworks that can account for the lossy nature of compaction.

Enterprise Production & Red-Teaming Blueprint

For enterprise production and red-teaming, it is crucial to implement robust compaction strategies that minimize information loss and prioritize constraint retention. This involves evaluating different compaction methods and thresholds to determine their impact on task performance and constraint retention. Additionally, it is essential to develop comprehensive evaluation frameworks that can account for the lossy nature of compaction and ensure the integrity of agentic tasks.

Technical FAQ (Granular & Non-Repetitive)

Conclusion

The study by Jeanne Salle et al. provides a comprehensive analysis of compaction strategies in long horizon agentic tasks. The findings highlight the need for robust compaction strategies that minimize information loss and prioritize constraint retention, as well as the need for comprehensive evaluation frameworks to ensure the integrity of agentic tasks. By addressing these limitations, we can enhance the performance and reliability of long horizon agentic tasks, paving the way for more advanced and efficient AI systems.