Do Think Tags Really Help LLMs Plan? A Critical Evaluation of ReAct-Style Prompting

We thank the reviewer for their thoughtful comments and feedback, and that they find our study effective, well-structured, and the motivation valuable with respect to future research in LLM prompting methods. We have tried addressing each of the concerns below:

$**Experiment analysis (W1a):**$ We thank the reviewer for this insightful question. In all our experiment settings, we note that LLMs replicate the exact steps as shown for the examples in the prompts. Hence, they do NOT output what ReAct claims as think tags if those tags are not present in the original prompt. We agree that this needs to be clarified clearly in the paper’s results, and thus, we have updated our Results section for lines 425-427 to further emphasize on this point.

$**Failure analysis (W1b):**$ We would like to highlight that these failures are due to LLMs being unable to correctly reason possibly due to the mismatch with the prompt examples as shown by our results in Table 3, and clearly not due to simpler syntactic mistakes. This is also true in the case even if one of the two prompt examples is the same as the problem query (see variation descriptions in Section 5.3).

$**Conclusion and Takeaways (W2):**$ In this work, we primarily intended to focus on the supposed usefulness of think tags, i.e., interleaved reasoning traces in multi-step text-based decision making problems. Our hypothesis behind the analysis was to give ReAct the benefit of doubt assuming that it is the reasoning trace (its content and location in the prompt) that leads to increased LLM performance on decision-making domains such as AlfWorld and WebShop. To recall, we note from Table 3 (RQ3) that changing the examples in the prompt drops the performance across multiple LLM models in the AlfWorld task, which is completely opposite to the case when we modify the location and content of the think tag in Table 1 (RQ1) and Table 2 (RQ2). While our results re-iterate on other findings regarding the role of examples in the few-shot settings, we wanted to systematically study and show how the reasoning traces, which is the primary claim behind ReAct and all the follow-up works that build on ReAct [1-6], is of practically no use and only leads to requiring prompt engineers include these reasoning traces in the examples.

To conclude, we believe that it will be helpful for future works to take these results into account, particularly when designing prompts for text-based decision-making problems, and benefit from avoiding putting any efforts into constructing reasoning traces but rather select the right examples for subsequent problems. We have included this point in our Conclusion section in lines 536-539.

[1]Yao Yao, Zuchao Li, and Hai Zhao. Beyond chain-of-thought, effective graph-of-thought reasoning in large language models. arXiv preprint arXiv:2305.16582, 2023.

[2]Noah Shinn, Beck Labash, and Ashwin Gopinath. Reflexion: an autonomous agent with dynamic memory and self-reflection. arXiv preprint arXiv:2303.11366, 2023.

[3] Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Ehsan Azarnasab, Faisal Ahmed, Zicheng Liu, Ce Liu, Michael Zeng, and Lijuan Wang. Mm-react: Prompting chatgpt for multimodal reasoning and action. arXiv preprint arXiv:2303.11381, 2023.

[4] Lluis Castrejon, Thomas Mensink, Howard Zhou, Vittorio Ferrari, Andre Araujo, and Jasper Uijlings. Hammr: Hierarchical multimodal react agents for generic vqa. arXiv preprint arXiv:2404.05465, 2024.

[5] Can Cui, Yunsheng Ma, Xu Cao, Wenqian Ye, and Ziran Wang. Receive, reason, and react: Drive as you say, with large language models in autonomous vehicles. IEEE Intelligent Transportation Systems Magazine, 2024.

[6] Yunjia Zhang, Jordan Henkel, Avrilia Floratou, Joyce Cahoon, Shaleen Deep, and Jignesh M Patel. Reactable: Enhancing react for table question answering. arXiv preprint arXiv:2310.00815, 2023.

We thank the reviewer for their thoughtful comments and feedback, and that they find our study effective, well-structured, and the motivation valuable with respect to future research in LLM prompting methods. We have tried addressing each of the concerns below:

$**Generalizability of results to other tasks (W1/Q1):**$ We would like to first highlight that we show experiments on both decision-making benchmarks that were originally used in ReAct. Furthermore, to recall the working of ReAct using the example of Put task (e.g., put some bottle on toilet) in the AlfWorld domain, the LLM needs to be provided the exact same action space via the examples to solve a subsequent Put task problem. As we show in our results for changing the examples in the prompt in Table 3, the LLM performance significantly diminishes. Note that there are cases where the examples can be from the Heat task (where the agent has to put some object in the microwave and heat it) that can be expected to work for a Put task problem (where the agent only has to put some object. The reason is that the action space for the Heat task already subsumes the action space required for solving the Put task, and yet, the LLM’s sensitivity is revealed by a significant drop in performance due to lack of generalizability. In this case, our argument is further pronounced that the LLM’s performance has little to no correlation with the interleaved reasoning traces (the example for Heat task contained the interleaved reasoning trace or the think tag which is similar to the think tag used for a Put task example).

$**Approach justification/LLM performance issues (W2/Q2):**$ We do include LLAMA-3.1-8B results for our RQ1 and RQ2 results on Webshop in Table 2, and observe a similar trend in results as we see in the black-box models from the GPT and Claude family. We agree that while these models may have different architectures and abilities, we tried establishing baseline ReAct results for each of these models and analyzed the gain or drop in performance across our different prompt variations. In this work, we do not aim to benchmark or analyze any single LLM’s reasoning abilities on decision-making tasks (to understand its limitations as rightly pointed by the reviewer), but rather intend to understand the robustness/brittleness of various LLMs with respect to different components in the ReAct-style prompting method for these tasks. Taking the reviewer’s suggestion, we have further updated our paper in lines 374-377 to reflect this point clearly for the readers.

$**Takeaways and workflow completeness (W3/Q3):**$ In this work, we primarily intended to focus on the supposed usefulness of think tags, i.e., interleaved reasoning traces in multi-step text-based decision making problems. Our hypothesis behind the analysis was to give ReAct the benefit of doubt assuming that it is the reasoning trace (its content and location in the prompt) that leads to increased LLM performance on decision-making domains such as AlfWorld and WebShop. To recall, we note from Table 3 (RQ3) that changing the examples in the prompt let alone drops the performance across multiple LLM models in the AlfWorld task, which is completely opposite to the case when we modify the location and content of the think tag in Table 1 (RQ1) and Table 2 (RQ2). While our results re-iterate on other findings regarding the role of examples in the few-shot settings, we wanted to systematically study and show how the reasoning traces, which is the primary claim behind ReAct and all the follow-up works that build on ReAct (see lines 126-130), is of practically no use and only leads to requiring prompt engineers include these reasoning traces in the examples. Regarding the different prompt and context window size, we do include the experiments (results shown in Table 3 for the column ‘All’) where we incorporate one example each from all AlfWorld tasks, thereby increasing the prompt length significantly as compared to the other settings. To include the point on the size of the context window clearly in our experiment section, we have now shown the context window sizes for each of the LLMs that we use for our results in lines 366-374.

To conclude, we believe that it will be helpful for practitioners and future works to take these results into account, particularly when designing prompts for text-based decision-making problems, and benefit from avoiding putting any efforts into constructing reasoning traces but rather select the right examples for subsequent problems. We have included this point in our Conclusion section in lines 536-539.

We thank the reviewer for their thoughtful comments and feedback, and that they find our study detailed and effective in analyzing ReAct. We have tried addressing each of the concerns below:

$**Citations to ICL literature (W1):**$ We do cite some of the relevant papers on ICL in our paper in lines 102-104 in Section 2 when talking about the different prompting strategies, and have also included the paper mentioned by the reviewer in lines 131-132. We would be happy to include any other specific references that the reviewer suggests can be helpful for our work.

$**Uselessness of think tags (W2):**$ In this work, we primarily intended to focus on the supposed usefulness of think tags, i.e., interleaved reasoning traces in multi-step text-based decision making problems. Our hypothesis behind the analysis was to give ReAct the benefit of doubt assuming that it is the reasoning trace (its content and location in the prompt) that leads to increased LLM performance on decision-making domains such as AlfWorld and WebShop. To recall, we note from Table 3 (RQ3) that changing the examples in the prompt let alone drops the performance across multiple LLM models in the AlfWorld task, which is completely opposite to the case when we modify the location and content of the think tag in Table 1 (RQ1) and Table 2 (RQ2). While our results re-iterate on other findings regarding the role of examples in the few-shot settings, we wanted to systematically study and show how the reasoning traces, which is the primary claim behind ReAct and all the follow-up works that build on ReAct (see lines 102-104), are of practically no use and only lead to requiring prompt engineers include these reasoning traces in the examples.

We thank the reviewer for their thoughtful comments and feedback, and that they find our study effective and the motivation valuable with respect to the ongoing research. We have tried addressing each of the concerns below:

$**Prompts’ space usage and terminology (W1):**$ We noted the reviewer’s concerns regarding the difficulty in following some parts of the paper, and have tried addressing the suggested changes for improved readability and easier understanding. Our intention behind including the important prompt changes in Section 4.1 was to provide an easy-to-reference example for each prompt variation. Based on the reviewer’s suggestion, we have updated the respective paragraphs (see blue text in Section 4.1 and Section 4.2) and refer the readers to Figure 2 which includes examples for all the variations in one place. Furthermore, gpt-3.5-turbo/gpt-3.5-turbo-0125 and gpt-3.5-turbo-instruct/gpt-3.5-instruct were used interchangeably in the paper. We included a description of the ‘Act’ baseline (originally used as a baseline in ReAct paper) in line 401 in our paper.

$**Clarification on environment dynamics (W2):**$ We would first like to clarify, based on the original paper on AlfWorld [1] that the environment is not dynamic, but partially observable which means that the agent may only find out if an object is present in a location or not after reaching that location. Moreover, as we note from our results in Table 1 (RQ1) and Table 2 (RQ2), the idea of having these reasoning traces for a couple of examples in the prompt pushes the LLM to explore the wrong rooms (as it is merely trying to replicate a similar flow of actions as shown in the example prompt). This further emphasizes our argument that these traces are not helpful. Coming back to the reviewer’s claim on the difficulty of providing all the information upfront, we surely would not be able to construct similar prompt variants for environments which are dynamic (for example, a modified AlfWorld). We have further included this as an additional comment in our paper in lines 211-213.

$**Clarification on RQ1 variants (W3):**$ The primary motivation behind our work is to point out the reliance on human-written prompts in ReAct. While we agree with the reviewer’s point that the LLM should generate a prompt given a user query, all the prompts used in ReAct originally require human effort to design prompts with examples that can potentially help the LLM agent to replicate the behavior on a new query problem. Our sensitivity analysis is focused on revealing this exact behavior, such that it can be easily understood which elements in the prompt indeed help with the downstream task.

$**Conclusion and takeaways (W4):**$ In this work, we primarily intended to focus on the supposed usefulness of think tags, i.e., interleaved reasoning traces in multi-step text-based decision making problems. Our hypothesis behind the analysis was to give ReAct the benefit of doubt assuming that it is the reasoning trace (its content and location in the prompt) that leads to increased LLM performance on decision-making domains such as AlfWorld and WebShop. To recall, we note from Table 3 (RQ3) that changing the examples in the prompt let alone drops the performance across multiple LLM models in the AlfWorld task, which is completely opposite to the case when we modify the location and content of the think tag in Table 1 (RQ1) and Table 2 (RQ2). While our results re-iterate on other findings regarding the role of examples in the few-shot settings, we wanted to systematically study and show how the reasoning traces, which is the primary claim behind ReAct and all the follow-up works that build on ReAct (see lines 126-130), is of practically no use and only leads to requiring prompt engineers include these reasoning traces in the examples.