Agent Trajectories Reveal Critical Intent-Execution Gaps in Frontier Model Behavior

Executive Summary & Core Discovery

The study of agent trajectories has emerged as a critical lens for understanding the true capabilities of frontier models. While pass@1 metrics provide a high-level snapshot of model performance, they fail to capture the nuanced differences in problem-solving strategies that emerge during actual agent execution. This research reveals that the intent-execution gap – the mismatch between what models intend and what harnesses execute – is a fundamental systems problem that can significantly impact agent performance.

Architectural Background & Theoretical Foundations

The research builds on the understanding that agent performance is not just a modeling problem but fundamentally a systems problem. The study introduces the concept of the intent-execution gap, which represents the mismatch between what the model intends and what the harness executes. This gap is crucial because it highlights how model assumptions about agent behavior may not align with the actual behavior of the execution environment.

To analyze this gap, the researchers developed the Simple Strands Agent (SSA), a customizable harness designed to find common patterns across different model families. The SSA framework allows for the generation and analysis of large-scale agent trajectories, providing insights into model behavior that traditional metrics cannot capture.

Benchmark Architecture & Evaluation Methodology

The evaluation methodology focuses on analyzing agent trajectories across four popular benchmarks: SWE-Pro, SWE-Verified, Terminal-Bench-2, and Deep-SWE. The researchers generated 154,000 trajectories using the SSA framework, which were then analyzed using code state-space representations. This approach allowed for the computation of solution distance, backtracking, edit frequency, testing activity, and phase transitions.

The analysis revealed that while frontier models may achieve similar pass@1 scores, their underlying problem-solving behaviors can vary significantly. This highlights the importance of examining agent trajectories for understanding model capabilities and potential limitations.

Empirical Findings & Granular Performance Analysis

The empirical findings demonstrate that the intent-execution gap can significantly impact agent performance. The analysis of 154,000 trajectories revealed that while frontier models may achieve similar pass@1 scores, their underlying problem-solving behaviors can vary dramatically. This is particularly evident in the code state-space representations, where differences in solution distance, backtracking, edit frequency, testing activity, and phase transitions become apparent.

In-Depth Analysis of Core Limitations & Vulnerabilities

The study identifies four core limitations that arise from the intent-execution gap:

1. Model Assumptions Not Aligned with Harness Behavior

The mismatch between model assumptions and harness behavior can lead to suboptimal agent performance. This is particularly evident in the code state-space representations, where differences in solution distance, backtracking, edit frequency, testing activity, and phase transitions become apparent.

2. Inefficient Exploration of Solution Space

The analysis reveals that frontier models may engage in inefficient exploration of the solution space, leading to increased backtracking and solution distance. This highlights the need for more sophisticated exploration strategies in agent design.

3. Over-Editing of Code Solutions

The study found that frontier models tend to over-edit code solutions, leading to increased edit frequency and solution distance. This suggests that models may benefit from more selective editing strategies to improve efficiency.

4. Under-Utilization of Testing Frameworks

The analysis demonstrates that frontier models may under-utilize testing frameworks, leading to reduced testing activity and potential issues with solution correctness. This highlights the importance of integrating testing into the agent design process.

Enterprise Production & Red-Teaming Blueprint

For enterprise deployment, the findings from this research provide a blueprint for minimizing the intent-execution gap and improving agent performance. By analyzing agent trajectories and code state-spaces, organizations can gain insights into model-level differences in problem-solving behavior and identify areas for improvement. This includes refining model assumptions, optimizing exploration strategies, implementing selective editing techniques, and integrating testing frameworks into the agent design process.

Technical FAQ (Granular & Non-Repetitive)