We sincerely thank the reviewer for the positive and thoughtful feedback. We're glad that you found our approach simple yet effective, and we appreciate your recognition of the paper’s originality, significance, and empirical contributions. To address your question regarding computational costs, we add additional analysis on the token count, number of forward passes, and training time of TTI. Below, we provide a detailed response to each comment.

---

Weakness 2: Compute Cost

Compute cost is brought up constantly in the paper, would it be possible to include them?

Thank you for this suggestion. While we reported token-level compute costs in Figure 3 (top), we agree that a more detailed breakdown for large-scale training would be helpful. We update Appendix F.3 to include the following table for the WebVoyager experiments (Table 3). All settings use Gemma 3 12B. Statistics computed on the training set. Training time includes both rollout collection and model update performed on 4 H100s for 10 iterations.

In summary, TTI improves success rate while requiring fewer training compute than fixed h=30 baselines, making it both more effective and efficient in our current setup.

---

Question 2: Fixed-Horizon Baseline Performance

The drop in performance in Figure 4 as training steps increase is bizarre. Can you explain why the performance drops on all domains for interaction 5 and 10?

We hypothesize that the drop in performance for short-horizon agents is due to:

• Early overfitting: With smaller horizons, the agent initially learns to exploit short paths but quickly overfits to local patterns that do not generalize well on the held-out set.
• Limited rollout diversity: These settings fail to expose the agent to exploratory trajectories or recovery strategies, leading to a plateau or even regression as training progresses.
• Reduced data coverage: Since each rollout is shorter, fewer states are visited overall, reducing the diversity of successful examples stored in the replay buffer.

We will add discussion of these dynamics in Section 5.1. The drops in fixed-horizon training actually motivate our curriculum approach. TTI (Figure 6) shows more stable learning because its curriculum helps mitigate this collapse by exposing the agent to longer interactions progressively, improving both generalization and robustness.

---

Question 1: Future Work

Do you plan to train your agent with a more sophisticated method than online behavior cloning? Long horizon tasks such as web browsing could benefit greatly from a method that performs credit assignment such as PPO.

Yes, we agree that more advanced RL algorithms such as PPO, GRPO, or Q-learning-based methods could offer improved credit assignment over online filtered behavior cloning, particularly for long-horizon tasks with sparse rewards. We chose to begin with a simple and robust baseline to isolate the effects of curriculum over interaction horizon, but we absolutely see value in applying policy-gradient-based methods or even value-based methods in future work.

As noted in Section 7, combining TTI with more advanced RL methods is a promising direction. In particular, using PPO could help stabilize training with longer horizons and enable more nuanced learning of exploration strategies. We plan to explore this in the future.

---

Weakness 1: Presentation

Figure 6 legend: Thank you for pointing this out. We have fixed the missing legend markers in the updated version.

Table 3 environment: Table 3 reports results on WebVoyager (we now explicitly clarify this in the caption and surrounding text).

Typo (“Comaprisons”): Fixed—thank you for catching this.

We thank the reviewer for their thoughtful feedback and for recognizing the empirical gains of our method. To address your concern about the number of experimental trials, we now include additional runs, demonstrating that our results are statistically consistent. We believe that these results should help address concerns regarding the efficacy of our approach.

At the outset, we would like to first address your concern that TTI is impractical because "extending the interaction horizon is a double-edged sword."

At a high level, a similar argument can be made about any expressive agent/policy architecture: using a larger model, more interaction steps, or longer reasoning chains, could in principle represent policies that increase the probability of risky behavior or incur higher latency and compute costs. However, recent advances in reasoning and agentic tasks have demonstrated that with the right training objectives and workflows, we can guide such expressive policies toward useful solutions rather than pathological ones, leading to improved generalization and performance scaling.

We developed TTI in the same spirit. The RL-based training used in TTI explicitly discourages risky or degenerate behaviors like submitting forms incorrectly or clicking fraudulent links, because these behaviors minimize reward (see our RL objective in Eq. 1). Thus, TTI allows the model to benefit from extended interaction only when it leads to better outcomes. Case studies in Section 6.2 also confirm that TTI agents learn strategic behaviors like backtracking and verification, not random exploration. In fact, pretty much like how in closed-domain reasoning, using reasoning improves model safety (e.g., see deliberative alignment from OpenAI), we believe that deliberative behaviors that use more interaction steps should only help an agent detect risk early on through exploration and planning, while still enabling it to solve the task eventually.

Like other test-time scaling techniques (e.g., long-chain CoT prompting in LLMs), TTI does introduce some overhead, but in exchange, it yields significantly more robust and capable behavior, especially on complex tasks where shorter horizons fall short. Indeed, just as long CoT traces have become widely adopted in LLM reasoning despite their compute cost, we view TTI as an analogous strategy for agentic settings, where interaction reveals new information critical for success. We therefore believe it would be unfair to reject our work based solely on presumed risks or overhead, particularly in light of the benefits.

Below, we address other comments in detail.

---

Weakness 3: Multiple Runs

Main results come from a single run, so robustness to random seeds remains uncertain at full scale.

Thank you for raising this point. First, we want to clarify that results in Section 4 and 5.2 are already averaged over three runs, as we noted in the caption of Figure 1 "results averaged over 3 runs." Since we only include the average numbers in the paper, we also provide the error bars below, which we will add to the appendix.

For large-scale RL training on WebVoyager and WebArena (Table 3 & 4), we note that the training tasks are over 100K, making rollout collection super expensive. Due to resource constraints, we thus prioritized domain and benchmark coverage over running multiple seeds. We note that recent work on web agents that are published in conferences, such as Learn-By-Interact (ICLR 2025), AgentOccam (ICLR 2025), WebRL (ICLR 2025), and Agent Workflow Memory (ICML 2025), all report single-run results on large-scale benchmarks due to the cost of training and evaluation. We agree that adding multiple seeds is valuable and will include this in the final version, though we expect our results and conclusions to be solid.

Finally, several other reviewers (kwcd, 13jF) independently noted the consistency of performance across 20+ domains and 2 benchmarks. Besides, our results can be reproduced with the released codebase and training details, and we plan to open-source checkpoints for verification. These all improve the reproducibility and robustness of our work.

---

Weakness 2: Curriculum Heuristic

The horizon-scheduling curriculum that tackles the training difficulty of longer interactions is still largely heuristic.

While our curriculum does not emerge from a theoretical derivation, it is grounded in well-established practices from both RL and LLM training rather than merely being ad-hoc heuristics. Specifically,

Horizon scaling has been shown effective in prior work such as CodeT [1] and Reflexion [2], though none of these works have proposed a formal training mechanism. Our horizon-based schedule mirrors established RL curricula [3, 4] that gradually increase task difficulty to improve stability under sparse rewards.

As for the specific curriculum schedule, our use of a multiplicative curriculum (Eq. 3) is not purely heuristic because: (1) multiplicative schedule is very common in existing reasoning works; (2) we study several design choices such as additive schedule as ablations. Both of them outperform baseline agents and test-time-only strategies like budget forcing or best-of-n prompting (Table 2 and Figure 3). However, the multiplicative schedule is empirically more effective, so we adopt it for later experiments; (3) on WebVoyager, TTI outperforms fixed-horizon baselines (Figure 4, 6). Such strong downstream results show the effectiveness of our curriculum design.

[1] CodeT: Code Generation with Generated Tests. ICLR 2023.

[2] Reflexion: Language Agents with Verbal Reinforcement Learning. NeurIPS 2023.

[3] WebRL: Training llm web agents via self-evolving online curriculum reinforcement learning. ICLR 2025.

[4] AutoWebGLM: A large language model-based web navigating agent. KDD 2024.

---

Question: Applicability to Risky Settings

Can TTI be used in general tasks and environments where each action is risky or costly and, if so, what changes would it need?

How can we keep extra actions from becoming too expensive or dangerous while still reaping the benefits of longer interaction?

Yes, TTI can be adapted to general environments where actions are risky or costly as future work. For example, we can introduce cost-aware curricula that assign higher weights to risky actions and optimize performance subject to total cost constraints, similar to safe RL frameworks [1]. We can also enforce risk-bounded exploration by limiting high-impact actions during early training, and gradually relaxing constraints. For example, in robotics, TTI could first allow only low-cost actions such as camera adjustments, then gradually introduce manipulation steps, with irreversible actions gated behind explicit confirmation.

As for balancing longer interaction and the quality of each action, our findings in the paper already suggest a natural division of labor between internal reasoning and external action: internal "thinking" for planning and verification, and external interaction for exploration and information gathering. Thus, we can develop hybrid test-time scaling, where agents learn to dynamically allocate compute based on environmental cost and task uncertainty to achieve robust performance gains. We will add the above discussion to Section 7.

These extensions are important directions for applying TTI to more real-world settings, but are beyond the current scope of our work. We think it would be unfair to reject our paper based on the presumed risks in diverse environments.

[1] A Review of Safe Reinforcement Learning: Methods, Theory and Applications, Gu et al., 2024.

Thank you for highlighting our work's clarity, careful experiments, and empirical insights. To address your questions regarding evaluation, we add new results showing how per-step thinking changes during TTI training and new baselines that use RL to scale per-step thinking. These results further showcase how TTI extends interaction and outperforms approaches that only scale per-step reasoning. Below, we elaborate on the findings and also discuss questions around related work, novelty, and broader applicability.

---

Question 1

in Fig 4, after 400 steps, the agent (h=20) stop increasing overall interaction horizon but still improves the performance. Could you clarify this and the saying in L248?

Fig 4 shows that interaction length saturates after 400 steps for h=20 agent, yet performance continues to rise. This is not a contradiction, rather, the agent learns to use its interaction budget more effectively, choosing better actions and exploring more strategically (e.g., searching for the correct item instead of clicking randomly). This motivates why we want to train agents with longer horizons during later stages of training.

As for L248, it is meant to motivate why we want to use shorter horizons during early stages of training. By "worse performance stemming from noisy credit assignment and slower learning", we are referring to the initial phase of the h=20 agent (red dotted line in Fig 4 left), where the performance first decreases because h=20 agent has not yet grasped the basic abilities to solve tasks efficiently and associates random exploration moves with high rewards. Employing an adaptive horizon schedule via TTI can prevent the initial decrease and improve overall learning efficiency. We'll add this clarification to Sec 5.1.

provide the thinking lengths

As suggested, we add two sets of results to show how thinking token changes during RL training. We show tables here but will include figures in the paper.

For WebArena (Fig 4), we compute the average token in the "Thought" section for h=5/10/20.

For the h=5 agent, per-step reasoning increases over training, likely because the agent must solve the task within a limited number of steps, requiring more deliberate decisions. In contrast, h=20 agent relies more on extended interaction and shows a decrease in per-step reasoning. This aligns with our findings on large-scale WebVoyager (see below).

For WebVoyager results (Fig 5), we show the following:

As training goes, the agent's trajectory grows longer (see Fig 5 left), but the number of tokens appearing in per-step reasoning actually becomes smaller. This implies that our agent is automatically learning to trade-off interaction for per-step compute in order to attain higher performance, and perhaps prevents any issues with overthinking each step.

---

Question 2

In Fig 6, the "check-again" strategy seems to be less effective to increase the trajecory lengths. Could you provide interaction times for the RL trained agent with "check-again"? If it fails to improve RL (well-)trained agents, how could we further increase the performance?

As suggested, we provide the interaction times for RL agent with check-again.

As you pointed out, the increase in interaction steps is smaller for well-trained agents than baseline agents. Fig 6 also shows that the benefit of check-again diminishes as the agent learns to self-reflect during TTI training. This shows TTI agents learn to internalize re-checking and interaction scaling behavior via curriculum RL.

To further increase interaction horizons for well-trained agents, we could consider new directions like hierarchical exploration (e.g., setting and pursuing sub-goals within a single rollout). These are future work beyond the current scope of TTI. We'll add a discussion to encourage exploring this in the community.

---

Weakness 3

RL training to incentivize longer thinking is not compared with RL training for TTI

As suggested, we add new baselines that use RL to scale per-step reasoning for the WebVoyager experiments. We tried two methods: (a) using a more detailed instruction for the "Thought" section, or (b) asking the agent to double check its action at each step by outputting two "Thought…Action…" blocks. We use a fixed horizon of 10.

Both (a) and (b) increase per-step thinking. (a) improves over the fixed h=10 baseline but still underperforms compared to TTI. This suggests that simply thinking more per step is not enough; TTI's ability to scale interaction allows the agent to actively explore the environment and gather information critical to task success, leading to stronger performance. Also, method (b) seems to hurt performance likely because it adds redundancy rather than improving exploration or decision quality. We’ll include the new results in the paper.

---

Question 3

observation history is only recent 3 steps, how does the agent handle long-term dependencies

As noted in Sec 3, the number of web page observations is kept to 3; our agent still has full access to all past actions. This action history can serve as a compressed task "memory". Also, TTI implicitly teaches the agent to revisit pages, backtrack, and gather missing information dynamically to improve long-term dependencies (see case studies in Sec 6.2).

---

Question 4

L127 mentions prior works "usually scale the number of thinking tokens at each step" without reference. What are the prior works? Could you compare TTI with these prior works under prompting and RL training?

To the best of our knowledge, prior to submission, there were no multi-step agent works that systematically scale per-step thinking tokens during training (most focus on single-turn reasoning tasks like math). Recent work like Kimi K2 explores this, but it appeared much after our submission. That said, we already compare to baselines that scale thinking tokens:

1. Muennighoff et al. for test-time budget forcing (see Sec 4.2)
2. Claude, OpenAI models, and others trained for internal CoT scaling (see Tables 3, 4)
3. New RL baselines for scaling per-step reasoning, which we include in this rebuttal (see response to W3).

If we are missing prior work, please let us know and we are happy to compare!

---

Weakness 1

core idea of scaling interaction horizons is intuitive and not novel

While we agree that TTI's core idea is intuitive, we view its simplicity as a strength, not a weakness. Crucially, our idea is still novel, as prior work (e.g., Reflexion, CodeT) has used longer interactions primarily at test time, either for evaluation or to inject human priors via prompting. In contrast, our goal is to build a systematic curriculum over interaction steps that trains agents via RL to acquire the ability to leverage longer interaction horizons to improve performance. For prior approaches, such ability is not explicitly optimized during training. While scripted retries or prompting tricks can still tackle a few extra tasks with longer test-time horizons, their agents lack the capacity to adaptively scale interaction, which TTI agents are capable of.

---

Weakness 2

20/30 steps is still relatively short for complex tasks

We note that a 30-step horizon is already 2-5x longer than typical task lengths in WebArena (~6 steps), Mind2Web (~8 steps), and UI-TARS-1.5 training data (~15 steps), and aligns with the standard 30-step limit used in most benchmark frameworks. This provides sufficient room for exploration while remaining compute-efficient. We agree that longer horizons may be necessary in other domains and we encourage researchers to adapt step limits to their specific settings.

---

Open Questions

How should TTI be applied to other tasks that require high costs for interaction and with non-recoverable states? If there is a place for thinking longer to be effective, how should TTI co-exist with thinking longer?

We think the following strategies might be worth exploring:

1. Simulated interaction or planning can be used to reduce the number of costly real-world steps. For example, we can develop hierarchical TTI that differentiates between cheap simulated interactions and expensive physical actions, allocating horizon accordingly
2. We can also develop risk-aware curricula that gradually expose agents to higher-cost actions, e.g., in robotics and in several games, you have the option of "going back and retrying", meaning there is a possibility of developing risk aware curricula

Akin to how long CoT reasoning is not broadly applicable to every domain, long interaction lengths are also not, but this does not undermine the need for long CoT reasoning models.

Furthermore, our results suggest that longer per-step thinking is best suited for local optimization, validation, or final decision verification, whereas external interaction is more effective for exploration, gathering missing information, and adapting to partial observability. We believe an important future direction is hybrid test-time scaling, where agents learn to allocate their compute budget between internal reasoning and external action based on the environment’s cost and the agent's uncertainty.

We sincerely thank the reviewer for the thoughtful and encouraging feedback. We appreciate your recognition of the originality and significance of our work. To address your questions regarding more visualizations, we will add new per-domain training curves to the paper. Below, we provide responses to each comment.

---

Weakness 1

Adding per-domain training curves (success rate vs. training steps) would be helpful.

This is valuable feedback that would strengthen our analysis significantly! Due to space constraints, we prioritized aggregate results and included training curves for representative domains (e.g., Amazon, HuggingFace) in Fig. 5 (right). However, we have now added per-domain success rate trends across training iterations for all WebVoyager domains in Appendix F.9 of the updated version and would be happy to highlight these further in the main paper.

Since figures cannot be shown in the response (and neither can links), we summarize key findings below. The domains exhibit distinct learning dynamics, which we broadly group into three categories:

• Fast-learning domains (e.g., BBC, Apple): Plateau after ~5–6 iterations.
• Moderate-learning domains (e.g., Allrecipes, HuggingFace): Improve steadily over 8–10 iterations.
• Prone-to-overfitting domains (e.g., GitHub, Amazon): Performance may decline in later stages, as the agent learns to explore more despite the test tasks being simple and not requiring exploration (see our response to Q1 below for further discussion).

These differences appear to stem from domain complexity and the base model’s familiarity with each domain. We believe these insights are an important step toward understanding when and where TTI is most beneficial, and they point to promising future directions like tailoring curriculum schedules to domain difficulty or agent priors.

---

Question 1

In the GitHub, ESPN, and Coursera domains in Table 3 the base Gemma 3-12B outperforms its TTI-trained counterpart. Do these domains share characteristics that explain the drop in performance when applying TTI?

This is an excellent observation. In our analysis (Section 6.2 and Appendix F.6), we found that these domains are likely to be better represented in the base model's pretraining data, so the zero-shot agent already performs pretty good. In these cases, TTI’s added exploration can occasionally lead to unnecessary detours, analogous to how reasoning models can overthink on simple tasks. For example, the TTI agent may query external search tools when a direct filter suffices (GitHub), or compare multiple courses when one stands out clearly (Coursera). These are recoverable inefficiencies of our approach, not failures of the method.

We view the performance in GitHub, ESPN, and Coursera positively and as motivation for developing a more adaptive system in the future. Akin to how reasoning models benefit from learning when to invoke long CoTs and when to produce a response directly (referred to as “dual mode thinking”), we can improve TTI by dynamically allocating interaction vs. per-step compute based on task instruction features. We view this as an important next step of our work, that can build upon the training protocols we introduce or distill our agent.

---

Question 2

The variant trained with a fixed horizon of h = 10 outperforms both the curriculum schedule and the h = 30 variant on several domains in Table 3. Why is this intermediate horizon especially effective for the Amazon, ArXiv, GitHub, and Coursera domains?

The fixed horizon of h = 10 performs well in domains like Amazon, ArXiv, GitHub, and Coursera because many tasks in these settings have short, structured workflows, e.g., "find a paper by title," "open a GitHub repo," or "enroll in a top-rated course." In such cases, a longer horizon may introduce unnecessary exploration that distracts from the optimal path, similar to overthinking in CoT models. However, this is not a limitation of TTI, but a reflection of its general-purpose design. While fixed h = 10 happens to be well-matched to simple and straightforward tasks, TTI is trained to adaptively scale horizons across diverse domains. As shown in Figure 5, TTI matches or exceeds h = 10 on most domains and excels in information-dense or partially observable settings like Allrecipes or Cambridge, where longer interaction is essential.

We believe future extensions of TTI could incorporate domain-aware horizon control to better align with task complexity. This is a future direction we plan to explore.