Measuring AI Ability to Complete Long Software Tasks

Thank you for your detailed review of our work! We appreciate that you found our work highly significant and original, and that you found our writing clear and methodology rigorous.

• Distinction between very short tasks: We agree that short tasks are difficult to measure, and we attempted to mitigate this by designing the SWAA baselining app for minimal friction. The median SWAA task has a stdev of a factor of 1.43 in its human baseliner time (i.e. median_{t \in tasks} exp(stdev_{all baseline times on task t}(log(human time))) = 1.43), much lower than the range of SWAA task lengths, which is about a factor of 20. Note a limitation of SWAA is that many short segments of intellectual work are not capturable in standalone tasks, so SWAA is still far from a representative distribution of 2-second or 15-second tasks.
• Differences between time horizons of different reliability: We agree that it’s interesting that the tasks model can complete very reliably are so much shorter than tasks they can complete 50% or 20% of the time, though we’ll note that this seems consistent with anecdotal evidence about how current frontier models can be competent on average but are not reliably so. Interestingly, we do observe that 80% horizons are improving at similar rates to 50% horizons (Figure 17), and we find this exponential increase more convincing than the exact time horizon figures. We’ll edit the paper to include more discussion about these observations.

Questions:

• Differences between SWAA and other baseliners: The baseline methodologies generally match the ways that humans interface with tasks in the real world but are not perfect. For SWAA, humans immediately type an answer, much like a short segment of SWE work. On HCAST and RE-Bench, the baseliner instructions are somewhat different-- "time to reach target score" vs "maximize score after 8 hours"-- so we wouldn't be surprised if the effective times differed by 25% or so. This error is swamped by variance derived from the limited number of long tasks, but a consistent methodology would be important for more precise time horizon estimates. We also do robustness checks, including fitting a trendline without SWAA (Figure 10) which is consistent with the main trend. In the camera ready, we’ll also have additional SWE-bench Verified data and analysis of GPQA and LiveCodeBench, which use different human baseline or time estimate methodologies. We also note that when forecasting, the trendline slope only weakly depends on the bias of human baseline data since the y axis spans orders of magnitude. A 2x bias in SWAA tasks will only affect the projected date of 1-month software AI by a few months.

• Speculation on future trend: Predicting the future is always difficult, but we can speculate on the four factors in section E.2. The effectiveness of agency training could push the trend in either direction at any time, but the prevalence of reward hacking behavior suggests it might cause faster progress on verifiable tasks than nonverifiable ones. Compute scaling will probably continue for between 2-5 more years before it slows down the trend. AI R&D automation will probably depend on the ML time horizon reaching some unknown threshold. As for AGI generalizing better than current AI systems, we still think this could happen but have little idea when. Our overall guess is that the future trend on verifiable tasks will be around every 3-5 months, and on realistic tasks about every 4-8 months, with large uncertainties. This will continue until either compute limitations, AI R&D automation, or AGI happen.

  As for the messiness of week+ long tasks, we think the low messiness of our dataset is an important limitation at all task durations. Most week-long tasks will have at least some messy segments, and the time horizon may be much lower on those subtasks leading to higher overall failure rates. It is rare for a software engineer to go a week without encountering poor specifications or resource limits. In some analysis which will be in the camera ready, OSWorld and WebArena (agentic computer use) time horizons are as low as 2-4 minutes, but tooling has improved faster than we thought with systems like ChatGPT Agent, so some types of messiness may have less of an impact.

Thanks for the detailed review of our work! We’re glad that you think our work provides a fresh perspective on evaluating LLMs and that you find our results informative and insightful.

• Human baseline sample size/population variance: We agree that the task lengths measured on different human populations will differ substantially, and we do some robustness checks related to this in Section 4. On SWE-bench Verified (Section 4.1), whose task lengths are estimates of high-context maintainer time, models have shorter time horizons. In our main analysis, the population of human baseliners is people from top-100 universities or equivalent aptitude, with several years of experience in their field (Section 2.2); this is consistent between HCAST, RE-Bench, and SWAA, with details of each in C.1.
• SWE-Lancer: Thanks for bringing up the SWE-Lancer paper! We’ll cite and discuss it in our work.
• Reducing variance or bias in measurement data: We have several ideas for how future work might increase measurement precision:
  • As we mention in Section 3.2, errors between time horizons are correlated, so relative comparisons have smaller error bars than absolute ones, e.g. a bootstrap test shows o3 is significantly above Claude 3.7 and above trend. We are also somewhat conservative with our CIs as they do 3-level hierarchical bootstrap, which some literature suggests overestimates variance.
  • Correcting for developer-specific effects (some humans are faster than others): not included because we had too few tasks with multiple human baselines to calibrate them accurately. The baseliner pool has different specialties for different tasks, so the tasks done by different baseliners were often disjoint.
  • Tasks could have subtasks like Cybench-- not done in this paper because the long tasks were taken from Rein et al. and Wijk et al., and the shorter SWAA tasks are only one step.
  • Tasks could be stratified by difficulty. This would require a new benchmark constructed to have task variants that are easier vs harder for models at the same length for humans, which would be difficult to keep realistic.
  • Item response theory methods could be applied to place more importance on more informative tasks. We rejected this for simplicity but it could be worthwhile in other contexts.
  • Overall we think there isn't enough statistical juice to be worth the complexity; as for future work we think it's best to collect human baseline data from real software development in the wild, which will have much lower marginal cost.

Thanks for your review!

We’d like to start by clarifying that our work is studying the ability of AI agents to perform agentic tasks – both RE-Bench and HCAST tasks require models to take multiple actions in series, and many of the agent runs use millions of tokens. We discuss these tasks in more detail in Section 3.1 and the ReAct-based agent scaffolds we used in Sections 3.3 and C.3. We agree that we could’ve been more clear that we’re studying AI performance on agentic tasks in the introduction and abstract; we’ll revise the terminology there to be more clear and consistent with the rest of the paper.

We also apologize for the typos on the number of models. In total, we evaluated 16 models (Figure 25), of which 12 advanced the state-of-the-art in terms of time horizon and 4 did not (and thus were ‘near-frontier’). We’ll edit the text to clarify this; thanks for pointing out these errors.

To address some other questions throughout the review:

1. Population of human baseliners: We agree that the task lengths measured on different human populations will differ substantially, and we do some robustness checks related to this in Section 4. On SWE-bench Verified (Section 4.1), whose task lengths are estimates of high-context maintainer time, models have shorter time horizons. We also measure the ratio of time taken by contractors vs maintainers on our internal PRs (Section C.2). In our main analysis, the population of human baseliners is people from top-100 universities or equivalent aptitude, with several years of experience in their field (Section 2.2); this is consistent between HCAST, RE-Bench, and SWAA, with details of each in C.1.
2. Task distribution vis-a-vis SWE development cycle: The HCAST (Rein et al) and RE-Bench (Wijk et al) papers have a breakdown of the categories each task falls under. HCAST is 30% MLE, 20% SWE, 20% cybersecurity, and 30% general CS-relevant reasoning, with subcategories given in Table 1 of the HCAST paper. RE-Bench is 7 long ML research engineering settings, each of which inherently require subskills like algorithmic thinking and debugging. As for SWAA, a breakdown between math, code completion, and other decisions is in appendix B.1.3. Overall, we have coverage of several SWE subfields and process steps, but lack the ability to measure some elements of such work that are inherently difficult to benchmark, like project management and code quality.
3. Choice of time horizon metric: We chose the time horizon metric due to its relevance to the role of AI in the future economy. When companies choose whether to automate a task, two important factors are the amount of human labor the AI can replace with minimal supervision, and the reliability of the AI. The time horizon metric combines these into a measure of performance that should relate to which tasks must still be done by human engineers vs. can be automated. Time horizon also has good explanatory power for the reliability of agents at different task lengths (Figure 4); the logistic fit is reasonably good and we find that agents’ 80% time horizons are roughly 5x shorter than their 50% horizons (Figure 17).
4. Other takeaways: Using time horizon, we observe that agent capability is more consistent with contractor time than experienced maintainer time (4.1, C.2), and that agents perform under their time horizon on more realistic tasks (H.2). By using human baselined time, we also quantify the degree to which agents can do impressive tasks but are unreliable on longer tasks (Figure 4). In the camera ready, we will also have an analysis of time horizons across 9 benchmarks in other domains like GPQA (scientific QA), AIME (competition math), RLBench (robotics), and OSWorld (graphical computer use). Models’ time horizons in other domains differ from their software time horizons, though we find similar rates of increase, and time horizon trends allow better forecasting of when automation in these other domains will be practical.

Thanks for the detailed review! We’re glad that you find our core time horizon metric novel and intuitive, and that you appreciate the breadth of tasks and models used in our empirical evaluations. We respond to each of your questions below.

1. Task complexity growth for longer task lengths: We find that a logistic curve against log-task-length space is the best way to model an agent’s success, and that the slopes of such logistic curves are fairly similar for each agent. If longer tasks were exponentially more complex, we would see steeper and steeper slopes, with agents able to consistently handle (say) 1-hour tasks but almost never 2-hour tasks. Instead we find that the 50% time horizons are roughly 5x longer than 80% time horizons (Section 3.2.1), at least on our data. This can perhaps be reconciled by a model where exponentially longer tasks have exponentially more steps, but the failure rate of agents is also decreasing exponentially over time.
2. Data contamination concerns: Since RE-Bench was released in late 2024, HCAST in 2025, and we developed SWAA privately, the problems are either private or past the training cutoff date of all models we evaluated (o3’s cutoff is June 2024). As for potential future models, some of the tasks we evaluate on are now public. HCAST authors claim 35/91 of the solutions in a subset of the 97 we used are potentially memorizable. RE-Bench has a public repo, but due to the small number of tasks we think it’s unlikely that a lab would train on it specifically. So overall, we think data contamination is not a concern on the models in this paper but may become relevant in the future. In the camera-ready, we will also include time horizons on LiveCodeBench, which is continuously updated to prevent contamination.
3. Gap between 50% and higher% task horizons: It is difficult to measure time horizons at reliability close to 100%, since one would need hundreds of independent samples and <<1% label noise to distinguish between 98% and 99% accuracy. However, we observe that 80% horizons are improving at similar rates (Figure 17), and 80% horizons are roughly 5x shorter than 50% horizons on all models, with the curve perhaps slightly steeper on more capable models (Figure 4).
4. Unseen/adversarial datasets and benchmarks: As discussed in Q2, all of the datasets are likely unseen to models. About other benchmarks, data were designed to be realistic rather than adversarial. However, we have looked at other benchmarks like SWE-bench Verified (Section 4.1), as well as GPQA Diamond, LiveCodeBench, and AIME (will be in camera ready). As discussed in the paper, some models have much shorter time horizons on SWE-bench Verified; on GPQA, 1-2x longer; this paper’s tasks are roughly in the middle.
5. Generalization to real-world software engineering performance: We perform several external validity checks (Section 4): measuring performance on SWE-bench Verified (4.1), subsets of HCAST that are more realistic (F.2), and on our internal PRs, which we manually reviewed for code quality as well as correctness (C.2). It also seems to be empirically true that models with higher time horizons also perform better on other SWE benchmarks, and anecdotally are more helpful for software engineers. To more definitively answer this question, in the future we’re hoping to study the software engineering performance of models at a higher level of realism, using real-world in-situ software engineering tasks done by professional software engineers.

Limitations: Thanks for referencing this work, we'll discuss and cite it! The referenced work is consistent with our qualitative analysis of failures (Table 2) where we observe that many failures of less capable agents are obvious, like repeated failed actions, whereas o1 makes mistakes that might be harder to supervise, like submitting answers without checking. We do observe that recent models like Claude 3.7 and o3 get near-perfect accuracy on tasks shorter than ~4 minutes (again see Figure 4), but we can’t rule out that performance of more capable models is lower on a different distribution of simple tasks.