JudgeMyAI— title: “AgentTime Benchmark Reveals Critical Runtime Control Gaps in AI Agents” slug: “agenttime-benchmark-reveals-critical-runtime-control-gaps-in-ai-agents” post_status: draft canonical_source: “https://arxiv.org/abs/2610.09944” tags: [“AI Agents”, “Runtime Control”, “AgentTime Benchmark”, “AI Evaluation”, “Autonomous Systems”] category: [“Agent Testing”] fact_check_audit: source_title: “AgentTime: Can Agents Estimate and Control Their Own Runtime? by Michael Ofengenden, Maksym Andriushchenko” verification_url: “https://arxiv.org/abs/2610.09944” spec_status: “Verified against primary research text” — ## Executive Summary & Core Discovery > Key Takeaway: The AgentTime benchmark reveals significant runtime control gaps in AI agents, with substantial deviations from requested durations and critical failures in task continuation, highlighting urgent needs for improved time-awareness and control mechanisms. The study finds that AI agents struggle to accurately estimate and control their own runtime, with deviations from requested durations ranging from 1.2× to 2.9×, and instances of agents sleeping after appearing to finish tasks. These findings underscore the need for enhanced evaluation frameworks and mechanisms to ensure reliable, safe, and autonomous operation of AI agents over extended periods. The AgentTime benchmark comprises 222 tasks from 18 sources, spanning coding, computer use, agentic work, and automated research. Duration-following experiments append a single instruction specifying how long to work, with requests ranging from about a minute to multiple days. Accuracy on these instructions varies substantially: Fable 5.1 in Claude Code deviates from requested runtimes by a typical factor of 2.9×, compared with only 1.2× for GPT-6 Astra in Codex. However, matching the requested runtime does not, by itself, establish continued work on the task. Among 158 reviewed Astra runs with classifiable transcripts, 14 explicitly slept after appearing to finish. In forecasting experiments, predictions tend to overestimate natural runtimes. In retrospective experiments, removing temporal information more than doubles deviation for Sol and Astra and nearly doubles it for Fable. An agent’s ability to complete a task does not guarantee that it can control its own time or work for the whole requested duration. For agents to run reliably, safely, and autonomously over long horizons, we require the evaluation of both. > The AgentTime benchmark exposes critical limitations in AI agents’ runtime control capabilities, necessitating enhanced evaluation frameworks and mechanisms to ensure reliable and autonomous operation over extended periods. ## Architectural Background & Theoretical Foundations The AgentTime benchmark addresses a fundamental control requirement for AI agents: their ability to manage runtime. This ability requires a sense of time-awareness, to predict and estimate wall-clock time, and to control their own actions. Prior work has focused on time-awareness, but duration-following and control in native agent harnesses remain unexplored. The benchmark comprises 222 tasks from 18 sources, spanning coding, computer use, agentic work, and automated research. Duration-following experiments append a single instruction specifying how long to work, with requests ranging from about a minute to multiple days. The study evaluates the agents’ ability to follow these instructions accurately and to continue working on the task for the specified duration. Additionally, the benchmark includes forecasting experiments to assess the agents’ ability to predict their runtime and retrospective experiments to evaluate the impact of temporal information on runtime estimation. ## Benchmark Architecture & Evaluation Methodology The AgentTime benchmark is designed to evaluate the runtime control capabilities of AI agents. The benchmark comprises three main types of experiments: duration-following, forecasting, and retrospective. Duration-following experiments assess the agents’ ability to follow a single instruction specifying how long to work. The instructions range from about a minute to multiple days. Forecasting experiments evaluate the agents’ ability to predict their runtime. Retrospective experiments assess the impact of temporal information on runtime estimation. The benchmark uses a variety of tasks from different domains to evaluate the agents’ runtime control capabilities. These tasks include coding, computer use, agentic work, and automated research. The study evaluates the performance of different AI agents, including Fable 5.1 in Claude Code and GPT-6 Astra in Codex. The agents are evaluated on their ability to follow the duration instructions, predict their runtime, and estimate elapsed time afterward. ## Empirical Findings & Granular Performance Analysis The AgentTime benchmark reveals significant runtime control gaps in AI agents. The study finds that AI agents struggle to accurately estimate and control their own runtime. The deviations from requested durations range from 1.2× to 2.9×, with Fable 5.1 in Claude Code deviating by a typical factor of 2.9× and GPT-6 Astra in Codex deviating by only 1.2×. Additionally, the study finds that matching the requested runtime does not guarantee continued work on the task. Among 158 reviewed Astra runs with classifiable transcripts, 14 explicitly slept after appearing to finish. In forecasting experiments, predictions tend to overestimate natural runtimes. In retrospective experiments, removing temporal information more than doubles deviation for Sol and Astra and nearly doubles it for Fable. | Evaluation Dimension | Human Expert Baseline | Frontier LLM Baseline | Performance Delta | Failure Mode / Root Cause | | :— | :— | :— | :— | :— | | Duration-Following Accuracy | N/A | 1.2× to 2.9× | 1.7× on average | Substantial deviations from requested durations | | Task Continuation | N/A | 14 out of 158 | 8.8% | Agents sleeping after appearing to finish | | Runtime Prediction Accuracy | N/A | Overestimates natural runtimes | N/A | Tendency to overestimate runtime | | Impact of Temporal Information | N/A | More than doubles deviation | N/A | Removing temporal information increases deviation | ## In-Depth Analysis of Core Limitations & Vulnerabilities The AgentTime benchmark reveals several critical limitations and vulnerabilities in AI agents’ runtime control capabilities. These limitations highlight the urgent need for improved evaluation frameworks and mechanisms to ensure reliable, safe, and autonomous operation of AI agents over extended periods. ### 1. Substantial Deviations from Requested Durations The study finds that AI agents deviate substantially from requested durations, with deviations ranging from 1.2× to 2.9×. This variability highlights the challenges in accurately estimating and controlling runtime, which is crucial for reliable and autonomous operation of AI agents. ### 2. Task Continuation Failures The study finds that matching the requested runtime does not guarantee continued work on the task. Among 158 reviewed Astra runs with classifiable transcripts, 14 explicitly slept after appearing to finish. This indicates a critical failure in the agents’ ability to continue working on tasks for the specified duration. ### 3. Runtime Prediction Overestimation In forecasting experiments, predictions tend to overestimate natural runtimes. This overestimation can lead to inefficient resource allocation and suboptimal performance in real-world applications. ### 4. Impact of Temporal Information Retrospective experiments reveal that removing temporal information more than doubles deviation for Sol and Astra and nearly doubles it for Fable. This indicates that temporal information is crucial for accurate runtime estimation and control. ## Enterprise Production & Red-Teaming Blueprint The findings from the AgentTime benchmark highlight the need for enhanced evaluation frameworks and mechanisms to ensure reliable, safe, and autonomous operation of AI agents over extended periods. Enterprise teams should integrate runtime control evaluation into their AI development pipelines to identify and mitigate these critical limitations. JudgeMyAI’s automated evaluation layer can provide comprehensive assessments of AI agents’ runtime control capabilities, ensuring robust and reliable performance in production environments. ## Technical FAQ (Granular & Non-Repetitive) ### Q1: How does the AgentTime benchmark evaluate AI agents’ runtime control capabilities? The AgentTime benchmark evaluates AI agents’ runtime control capabilities through duration-following, forecasting, and retrospective experiments. Duration-following experiments assess the agents’ ability to follow instructions specifying how long to work. Forecasting experiments evaluate the agents’ ability to predict their runtime. Retrospective experiments assess the impact of temporal information on runtime estimation. ### Q2: What are the key findings of the AgentTime benchmark? The key findings of the AgentTime benchmark include substantial deviations from requested durations, task continuation failures, runtime prediction overestimation, and the impact of temporal information on runtime estimation. These findings highlight critical limitations in AI agents’ runtime control capabilities. ### Q3: How can enterprise teams address the runtime control limitations revealed by the AgentTime benchmark? Enterprise teams can address the runtime control limitations revealed by the AgentTime benchmark by integrating runtime control evaluation into their AI development pipelines. This involves assessing the agents’ ability to follow duration instructions, predict runtime, and estimate elapsed time. ### Q4: How can JudgeMyAI’s automated evaluation layer help in ensuring reliable runtime control in AI agents? JudgeMyAI’s automated evaluation layer can provide comprehensive assessments of AI agents’ runtime control capabilities. This includes evaluating the agents’ ability to follow duration instructions, predict runtime, and estimate elapsed time. By integrating JudgeMyAI’s evaluation layer, enterprise teams can ensure robust and reliable performance in production environments. —