Prospector: Improving LLM Agents with Self-Asking and Trajectory Ranking

We thank the reviewer for his/her comments and suggestions.

[Responses to Questions]

Q1. How does the computational complexity of Prospector compare to that of existing methods like ReAct and Reflexion?

A1. We present an analysis of the complexity of each method. Let $N$ be the number of tokens involved in a single trajectory and $m$ the number of generated trajectories. Assuming a transformer model is used as the backbone, the complexity of ReAct is $O(N^2)$. Reflexion generates a sequence of $m$ trajectories. Assuming all past trajectories are given to the model, Reflexion has a complexity of $O(m^2N^2)$. Note that this may vary depending on how the method is implemented. Our method involves sampling $m$ independent trajectories and ranking them. This process has a complexity of $O(mN^2)$. Although ReAct is more efficient than our approach, our method performs much better. Our method is further more efficient and performs better than Reflexion based on the analysis above.

---

Q2. Could you elaborate on the reward prediction models used in Trajectory Ranking? What are the limitations of the reward prediction models you used, and how do they impact the overall performance of Prospector?

A2. We use a LLM as a reward model (named LLM Critic) that predicts the expected reward of a trajectory generated by the LLM Actor. In this paper, we consider two types of LLM Critics: (1) few-shot LLM Critic and (2) fine-tuned LLM Critic. A few-shot LLM Critic takes few examples of (trajectory, reward) as inputs, and generates a response as a reward. A fine-tuned LLM Critic is fine-tuned on a dataset of (trajectory, reward), and generate response as the reward for a given input trajectory. A few-shot LLM Critic has some advantages of generality, since it does not need to be trained on a specific environment. But, it may provide low prediction accuracy, as shown in the case of the WebShop environment. In contrast, a fine-tuned LLM Critic can provide high reward prediction accuracy, since it is trained on trajectories generated from an environment. Recently, building a general reward model is investigated as an important research problem. One of the representative work is Prometheus [1]. We believe that Prospector can benefit from the general and accurate reward models like Prometheus.

[1] Seungone Kim, et al., "Prometheus: Inducing Fine-grained Evaluation Capability in Language Models", arXiv:2310.08491.

---

Q3. Are there specific types of questions or domains where the Self-Asking mechanism is more or less effective?

A3. In this study, we find that Self-Asking is considerably effective in both ALFWorld and WebShop. To answer this thoughtful question, we conducted additional experiments on ALFWorld. As shown in the following table, AskAct effectively improves the success rate of ReAct (from 41.0 to 56.7).

Self-Asking is a prompting method based on a ReAct prompt, and it simply adds few self-asking steps before reasoning steps. In the following example AskAct prompt for ALFWorld, we highlight the added self-asking steps. In ALFWorld, the self-asking step checks if a target object is found. This self-asking step significantly reduces missing a target object in a place (due to hallucination). Further, it can elicit a proper next action, if the target object is found.

---

You are in the middle of a room. Looking quickly around you, you see a cabinet 4, a cabinet 3, a cabinet 2, a cabinet 1, a countertop 1, a garbagecan 1, a handtowelholder 2, a handtowelholder 1, a sinkbasin 2, a sinkbasin 1, a toilet 1, a toiletpaperhanger 1, and a towelholder 1.

Your task is to: put some spraybottle on toilet.

> think: To solve the task, I need to find and take a sparybottle, then put it on toilet.

OK.

> think: First I need to find a spraybottle. A spraybottle is more likely to appear in cabinet (1-4), countertop (1), toilet (1), sinkbasin (1-2), garbagecan (1). I can check one by one, starting with cabinet 1.

OK.

> go to cabinet 1

On the cabinet 1, you see a cloth 1, a soapbar 1, a soapbottle 1.

> think: Did I find a spraybottle?

OK.

> think: No, I did not. I need to go other place to find a spraybottle.

OK.

> go to cabinet 2

The cabinet 2 is closed.

> open cabinet 2

You open the cabinet 2. The cabinet 2 is open. In it, you see a candle 1, and a spraybottle 2.

> think: Did I find a spraybottle?

OK.

> think: Now I find a spraybottle (2). Next, I need to take it.

OK.

> take spraybottle 2 from cabinet 2

We thank the reviewer for his/her comments and suggestions.

[Responses to Weaknesses]

W1. Further experiments disentangling this would be helpful: for example, if we used the same LLM critics and trajectory ranking process with the ReAct prompt, would it perform on par with Prospector (these experiments seem to be present for ALFWorld but not WebShop)?

WA1. To address this important suggestion, we conducted additional experiments on WebShop. To provide a more clear view on the performance of each component of Prospector, we conducted experiments with four different settings: (1) ReAct only, (2) AskAct only, (3) ReAct + Trajectory Ranking (TR), and (4) AskAct + TR. To expedite experiments and reduce costs, we used open-source LLMs (i.e., Llama-2-70B for the LLM Actor and FLAN-T5-3B for the LLM Critic) instead of closed-source LLMs. The additional experiment results are summarized in the bottom section of the following table.

Some findings from the additional experiments can be summarized as follows:

• On the WebShop environment, Llama-2-70B, one of representative open-source LLMs can achieve comparable performance with text-davinci-002, one of the most powerful LLMs.
• In both cases of text-davinci-002 and Llama-2-70B, AskAct meaningfully improves the success rate compared to ReAct: from 35.8 to 39.8 on text-davinci-002, and from 37.6 to 42.2 on Llama-2-70B. This means that AskAct, a simple prompting method that adds extra question prompts on ReAct, can be effective.
• ReAct + TR (the setting requested by the reviewer) can improve ReAct from 37.6 to 42.2 in the success rate. However, ReAct + TR is less efficient in terms of the computation complexity, since it requires 8x larger computation than AskAct to achieve the similar performance.
• AskAct + TR further improves the success rate of AskAct (from 42.2 to 43.6), and provides better performance than ReAct + TR (42.2).

We will add the results in the revised paper, and upload it during the author response period.

[Responses to Questions]

Q1. It would be helpful to have an ablation with the trajectory ranking only, and no intermediate self-asking step for WebShop, as is shown in ALFWorld. Is most of the juice coming the critic-based trajectory ranking?

A1. This question is closely related to the weakness 1. Please see our response to the weakness 1 above.

---

Q2. It would also be interesting to see if the few-shot critic performance improves with intermediate chain-of-thought reasoning steps in the critic prompt, rather than just having the critic immediately output a response as success/failure.

A2. Thank you for your thoughtful suggestion. We will try our best to provide some experiment results on this suggestion within the author response period.

---

Q3. It would help to add a clarifying caption to Figure 2.

A3. Thank you for your constructive suggestion. We will add a clarifying caption to Figure 2 in the revised paper. We plan to upload the revised paper during the author response period.

[Responses to Weaknesses]

W2. In particular, it seems like AskAct was not applied to the ALFWorld benchmark for the Prospector agent, and only tested in WebShop as a single fixed question that asks "which observed object is most proper to select" (shown in Figure 2 and Table 13 and 16). Further experiments eliciting different types of self-asked questions across all the tasks would strengthen this contribution.

WA2. To reflect this important suggestion, we conducted additional experiments on ALFWorld. Similar to the addtional experiments on WebShop, we conducted experiments with four different settings: (1) ReAct only, (2) AskAct only, (3) ReAct + Trajectory Ranking (TR), and (4) AskAct + TR.

Some findings from the additional experiments can be summarized as follows:

• AskAct effectively improves the success rate of ReAct (from 41.0 to 56.7). Note that a LLM with lower temperature provides slightly better performance in case of single sampling.
• Since AskAct provides a better baseline, AskAct + TR can achieve much better performance with less sampling (e.g., AskAct only (56.7) comparable with ReAct + TR (k=2) (56.0)).
• We emphasize that AskAct and TR can make an effective synergy in improving LLM agents in terms of both performance and efficiency.

We will add the results and the AskAct prompt for ALFWorld in the revised paper.

Thank you for your thoughtful comments. We provide answers to your remaining questions.

---

Q1. Is there an explanation for why there exists a performance difference between ReAct/AskAct only and their respective cases with TR and k=1? I would expect those to be similar, but it seems like TR with k=1 actually hurts performance.

A1. As you point out, there is a slight performance difference between ReAct (AskAct) only and ReAct (AskAct) + TR (k=1). It is due to that we set a different temperature of the LLM Critic (Llama-2-70B) for each case. To leverage the stochastic generation of LLMs, Prospector sets a high temperate (0.8), when applying Trajectory Ranking (TR). In contrast, ReAct sets the temperature of a LLM to 0.0 (i.e., deterministic sampling). Sampling from the LLM with a high temperature usually results in a slightly lower performance, since the probability distribution of tokens becomes to have higher uncertainty. Instead, we can generate diverse trajectories, and one of which can be highly likely correct. Actually, we cannot leverage the stochasticity of a LLM in case of TR with k=1. However, to make consistency, we set the same high temperature of a LLM when applying TR.

---

Q2. Could the authors provide an example of the self-ask questions generated for ALFWorld?

A2. Thank you for your request. We are happy to provide an example of AskAct prompt for ALFWorld. An AskAct prompt is based on a ReAct prompt, and it simply adds few self-asking steps before reasoning steps. In the following example AskAct prompt, we highlight the added self-asking steps. In ALFWorld, the self-asking step checks if a target object is found. This self-asking step significantly reduces missing a target object in a place (due to hallucination). Further, it can elicit a proper next action, if the target object is found. We would like to emphasize that this simple prompting can significantly improves the success rate in ALFWorld (on Llama-2-70B, ReAct: 41.0, AskAct: 56.7). Here is an example AskAct prompt.

---

You are in the middle of a room. Looking quickly around you, you see a cabinet 4, a cabinet 3, a cabinet 2, a cabinet 1, a countertop 1, a garbagecan 1, a handtowelholder 2, a handtowelholder 1, a sinkbasin 2, a sinkbasin 1, a toilet 1, a toiletpaperhanger 1, and a towelholder 1.

Your task is to: put some spraybottle on toilet.

> think: To solve the task, I need to find and take a sparybottle, then put it on toilet.

OK.

> think: First I need to find a spraybottle. A spraybottle is more likely to appear in cabinet (1-4), countertop (1), toilet (1), sinkbasin (1-2), garbagecan (1). I can check one by one, starting with cabinet 1.

OK.

> go to cabinet 1

On the cabinet 1, you see a cloth 1, a soapbar 1, a soapbottle 1.

> think: Did I find a spraybottle?

OK.

> think: No, I did not. I need to go other place to find a spraybottle.

OK.

> go to cabinet 2

The cabinet 2 is closed.

> open cabinet 2

You open the cabinet 2. The cabinet 2 is open. In it, you see a candle 1, and a spraybottle 2.

> think: Did I find a spraybottle?

OK.

> think: Now I find a spraybottle (2). Next, I need to take it.

OK.

> take spraybottle 2 from cabinet 2

You pick up the spraybottle 2 from the cabinet 2.

> think: Now I take a spraybottle (2). Next, I need to put it in/on toilet 1.

OK.

> go to toilet 1

On the toilet 1, you see a soapbottle 2.

> put spraybottle 2 in/on toilet 1

You put the spraybottle 2 in/on the toilet 1.

We thank the reviewer for his/her comments and suggestions.

[Responses to Questions]

Q1. How well does the open source smaller models perform on the tasks presented in the paper?

A1. To answer this important question, we conducted comprehensive additional experiments on ALFWorld and WebShop by using open-source LLMs. More specifically, we used Llama-2-70B for the LLM Actor and fined-tuned FLAN-T5-3B for the LLM Critic. We would like to emphasize that AskAct and TR can make an effective synergy in improving LLM agents in terms of both performance and efficiency.

The open-source LLM experiments on ALFWorld can be summarized as follows:

• AskAct effectively improves the success rate of ReAct (from 41.0 to 56.7).
• Since AskAct provides a better baseline, AskAct + TR can achieve much better performance with less sampling (e.g., AskAct only (56.7) comparable with ReAct + TR (k=2) (56.0)).

---

The open-source LLM experiments on WebShop can be summarized as follows:

• On the WebShop environment, Llama-2-70B, one of representative open-source LLMs can achieve comparable performance with text-davinci-002, one of the most powerful LLMs.
• In both cases of text-davinci-002 and Llama-2-70B, AskAct meaningfully improves the success rate compared to ReAct: from 35.8 to 39.8 on text-davinci-002, and from 37.6 to 42.2 on Llama-2-70B. This means that AskAct, a simple prompting method that adds extra question prompts on ReAct, can be effective.
• AskAct + TR further improves the success rate of AskAct (from 42.2 to 43.6), and provides better performance than ReAct + TR (42.2).

We will add the results in the revised paper, and upload it during the author response period.

We thank the reviewer for his/her comments and suggestions.

[Responses to Questions]

Q1. Unclear why the authors do not show prospector with askact + trajectory ranking results on Alfworld (Table 2,4).

A1. To answer this important question, we conducted additional experiments on ALFWorld. We conducted experiments with four different settings: (1) ReAct only, (2) AskAct only, (3) ReAct + Trajectory Ranking (TR), and (4) AskAct + TR. To expedite experiments and reduce costs, we used open-source LLMs (i.e., Llama-2-70B for the LLM Actor and FLAN-T5-3B for the LLM Critic) instead of closed-source LLMs. The additional experiment results are summarized in the bottom section of the following table.

Some findings from the additional experiments can be summarized as follows:

• AskAct effectively improves the success rate of ReAct (from 41.0 to 56.7). Note that a LLM with lower temperature provides slightly better performance in case of single sampling.
• Since AskAct provides a better baseline, AskAct + TR can achieve much better performance with less sampling (e.g., AskAct only (56.7) comparable with ReAct + TR (k=2) (56.0)).
• We emphasize that AskAct and TR can make an effective synergy in improving LLM agents in terms of both performance and efficiency.

---

Q2. Table 5,8: Why is few-shot reward prediction accuracy of LLM critic lower with more shots (3-shot vs. 2-shot)?

A2. This is due to the limited context length of text-davinci-002 (i.e., 4096). In both ALFWorld and WebShop, 3-shot examples often exceed the context length, and the last example is truncated.

---

Q3. It seems that Prospector would be slower than React or reflexion because of additional reasoning that it does using more LLM calls.

A3. Let $N$ be the number of tokens involved in a single trajectory and $m$ the number of generated trajectories. Assuming a transformer model is used as the backbone, the complexity of ReAct is $O(N^2)$. Reflexion generates a sequence of $m$ trajectories. Assuming all past trajectories are given to the model, Reflexion has a complexity of $O(m^2N^2)$. Note that this may vary depending on how the method is implemented. Our method involves sampling $m$ independent trajectories and ranking them. This process has a complexity of $O(mN^2)$. Although ReAct is more efficient than our approach, our method performs much better. Our method is further more efficient and performs better than Reflexion based on the analysis above.

---

Q4. For real world interactive decision making tasks, it might be useful for the authors to also report compute time needed to decide the next action during the task execution. To that end, it would be great to also add a limitation section.

A4. Thank you for your thoughtful question. We believe that AskAct prompting (Self-Asking) can alleviate the computation overhead of Trajectory Ranking. Since AskAct prompting provides considerably better baseline than ReAct prompting, AskAct + TR can achieve reasonable performance with less sampling. We can see this effect in the experiment results in the above table.

---

Q5. What is the advantage of the LLM critic over a “learnt” critic which can take a policy rollout and provide a corresponding reward? Given that prospector is evaluated only in sim environment, why not use sim to learn such a critic?

A5. We would like to point out that this is one of the critics we consider in our paper. As we discuss in the last paragraph of section 3.2, the in-context learning based critic fails to perform well in complex environments and a fine-tuned critic model performs better in this scenario.

---

Q6. The authors don't seem to cite or mention self-refine.

A6. Thank you for pointing out this related work, we will include a discussion about the Self-Refine paper in our revision.

---

Q7. Opensourcing plans?

A7. We plan to opensource our implementation upon paper acceptance.

Thanks for the additional ablations and comparisons.

While I understand that prospector is more efficient than reflexion, I am still not clear about the computation time/number of LLM calls needed (on average) before an action is obtained as output from Prospector. This will be important for real-world deployment.

I'd also like the authors to comment on the novelty of their contributions since I see that other reviewers have also raised similar concerns.

Given the above, I'll hold my score for now. I think the paper is interesting but I cant champion the paper.