Self-Challenging Language Model Agents

Thank you for your review and feedback on the paper. To address your concerns, we have collected additional experiment results with direct evidence of performance improvements and explain the dynamics between challenger and executor.

Direct evidence that filtering leads to performance improvements

To address your concern, we have included an explicit study comparing the model performance with and without CaT filtering. Specifically, we considered the ablation “SCA no filtering” that prompts the task challenger to interact with the environment and directly generate the instruction and evaluation function. The only difference of “SCA no filtering” from “SCA” is the lack of the CaT filtering mechanism. We generate 800 tasks and rollout 12K agent trajectories for each method. As shown in the following table, while SCA no filtering might have got some positive learning signals from the synthetic tasks to improve pass@4 performance, we found that the generated instructions and evaluation functions are quite noisy without filtering (as shown in the human analysis in Figure 4) and resulted in sub-optimal performance compared to SCA, e.g. pass@1 drops from 13.0 to 8.9. This shows that the filtering proposed in our paper is important.

Tau-Bench Retail

Dynamics between challenger and executor

We respectively disagree that the inclusion of an example solution in CaT limits the potential of self-improvement. In SCA, self-improvement can happen in at least two different ways: Many tasks are easier to verify than proposing a solution (https://www.jasonwei.net/blog/asymmetry-of-verification-and-verifiers-law) so it might be easier for the agent to come up with an evaluation function than proposing a solution. Once the task challenger has an evaluation function, it can try to “guess” the solution even if it does not know how to solve the problem. Thanks to the filtering mechanism in CaT, the wrong guesses are filtered and the valid tasks that remain can be used to reinforce the correct guesses when the same model serves as the executor. In this way, the executor keeps practicing by being rewarded for correct guesses until eventually it knows how to solve the problem and does not need to guess any more. Hence, the example solutions help it find what it can improve. Hindsight Experience Replay (https://arxiv.org/abs/1707.01495) has historically been a successful self-improvement method over many applications. The core idea is that the agent might accidentally hit some goals through explorations and we can use the goals that the agent accidentally hits as “hindsight experience” to improve the goal-reaching capability of the agent. SCA can operate in the same way as Hindsight Experience Replay. In the retail environment, the challenger constructing the task might randomly explore some user information in the database and try to run some commands such as exchanging a water bottle to a different size. If any one of the commands succeed, a confirmation message will be returned saying that the water bottle has been exchanged successfully. Based on the confirmation, the challenger knows that some changes have been made and can generate a user instruction in hindsight such as exchange the water bottle for John. But completing the task in the forward way might be much harder where the agent needs to first intentionally figure out the user id for John, and use the user id to find the transaction that contains a water bottle, and checks what sizes are available.

There are some writing inconsistencies, such as “Taubench” and “taubench”.

Thanks for spotting those inconsistencies, we will fix them in the next version.

Thank you for your review and positive feedback on the paper. We address your concerns below:

The experiments were conducted with only one model scale and one model architecture, which raises concerns about the generalizability of the proposed approach.

Because of compute and GPU memory constraints, we were only able to perform experiments at 7B scale. However, in our empirical results, SCA has been shown to improve agent performances across different environments (Table 1), across distillation and self-improvement settings (Table 1), across data scales (Figure 6), and across RL loss choices (Figure 3). We believe that these experiments already provide evidence for the generalizability of SCA.

The discussion of failure cases is insufficient.

Because of space constraints, we have deferred additional analysis of failure cases to the appendix (Appendix B and H, including examples in Figs 16,17, 18 and 19) and also referred to them in a dedicated Limitations Section (Section 6). Figure 4 shows that there is still a non-trivial percentage of False Negative samples and Appendix B demonstrates that there is still a sub-optimality gap between synthetic tasks from SCA and manually crafted tasks. We hope the discussion of these limitations can provide directions for future research. In the camera ready version with additional one page space we can move more of this discussion to the main paper.

More comparisons with existing methods in terms of computational cost are needed to demonstrate the practicality of the proposed approach.

The computational cost of SCA is comparable to other self-improvement methods like PAE because the main computation overhead lies in collecting agent rollouts to perform RL instead of the task proposals. We will provide more details on this in the camera ready version.

In this work, I have only observed improvements to the executor, raising the question of whether this work might encounter similar issues.

We also believe that such self-play would be a very promising future direction so that the executor and challenger can improve together. In our experiments, we did observe that the executor started to saturate on the tasks the challengers can offer. We believe this is an exciting research direction for future work.

Thank you for your review and positive feedback on the paper. We address your concerns below:

It is unclear whether SFT loss can be better in Self-improvement loss (RL algorithm).

We believe that there is a misunderstanding here. While indeed we adopted REINFORCE as our primary RL algorithm for the self-improvement setting, because of the reward structure of CaT in our experiments it is actually equivalent to SFT. As explained in Line 166-168, “Because of our reward structure with 0/1 outcome reward, this RL optimization objective is essentially equivalent to performing supervised finetuning (SFT) on successful trajectories (i.e. Rˆ 167 cˆ(sT , aT ) = 1) only, i.e. Rejection Fine-Tuning”. Importantly, however, comparisons of more RL losses – DPO, PPO and GRPO – can be found in Figure 3.

What are the possible ways to prevent these risks of false positives?

An important component of SCA is generating failure cases when creating the task – which we have found to significantly reduce false positives, see Figure 4. When the model generates failure cases, it can try to come up with as many corner test cases and failure modes as possible. CaT validation procedure ensures that the final tasks pass the tests of these corner test cases, so that during agent rollouts those corner cases will not be accidentally classified as success. Intuitively, the more corner test cases there are, the more robust the task should be against these false positives.

How about the reward is 1 and verification is wrong?

In our understanding, for the case of the reward is 1 and verification is wrong seems to mean that the agent got a reward 1 but it should get a reward of 0, meaning that the agent did not finish the task. This case seems to be included in "Agent did not complete the task but reward is 1", studied in Fig 4 and Sec 5.3 (see also Appendix C for annotation details).

It seems hard to conclude that unsuccessful are useful from experiments in Table 3.

Indeed, there is a confounding factor in this appendix experiment that there are only about 30% successful trajectories compared to all trajectories, if we keep the number of interactions the same for fair comparison. As a result, we refrained from making general conclusions in the appendix and only report this experiment result as an observation that we have found when the performance of the teacher model is significantly better than the student model. Further study of these aspects in future work would be good!

The plot in Figure 6 is not smooth. Are multiple runs in the experiments?

Because of the compute constraint and the fact that we need to train a separate model for each individual data point, we were unable to perform multiple runs to produce a smoother curve. However, from the 16 data points that we have gathered in Figure 6 with different numbers of training trajectories and training tasks, it already underscores the effectiveness of scaling the number of tasks through SCA compared to solely scaling the number of trajectories with the same tasks.

Thank you for your review and positive feedback on the paper. We address your concerns below:

The implementation details of the CaT architecture need further clarification.

Yes, if the task challenger fails to construct a valid task, it will receive feedback from the environment. If the task challenger fails to construct a valid task within 10 turns, this trajectory where the proposer interacts with the environment will be filtered out with no valid task. Indeed we noticed in our experiments that sometimes this can be as low as a 5% valid task construction rate for the task challenger as shown in Figure 4, but CaT still leads to significant self-improvement results.

Because of the space constraints, a lot of additional experiment details were deferred to the appendix. For instance, Appendix A includes additional environment-specific implementations of CaT and Appendix I provides an example of the task challenger interaction trajectory. We will move more explanations into the main text in the camera ready version using the extra page of space.

It needs to be compared with other self-improvement framework, such as Self-Adapting Language Model

In Table 1, we do have comparisons with the state-of-the-art self-improvement framework for agents PAE at the time of submission, and we have shown the advantage of SCA. While Self-Adapting Language Model is also related to self-improvement, it is released on June 12th – after the submission date – and it has not been tested in the agent domain.