Enhancing Language Model Agents using Diversity of Thoughts

We thank the reviewer for their valuable feedback. We address the raised weaknesses and questions below.

W1. We thank the reviewer for raising this point. We will clarify in the paper that the ablation you suggest is already captured by the difference between our two variants, DoT and DoT-Bank, which isolates the contributions of each module. Specifically:

1. DoT utilizes the Diverse-Reflections module and achieves an average gain of ~4.5% over the best competing method across all datasets and LLMs.

2. DoT-Bank incorporates the task-agnostic memory constructed during Phase-1 and further improves performance by an average of ~4.4%.

These results demonstrate that both components contribute independently to the performance gains, with their combination offering the highest improvements.

The key innovation lies in identifying and addressing the issue of repetition in naive reflections, which we quantify in the table below. Repetitive prompting fails to improve model performance, while our Diverse-Reflections method demonstrably generates more varied outputs, as evidenced by lower pairwise cosine similarity scores (quantified using the all-Mini-LM-v6 model from SentenceTransformers). These metrics are provided for HumanEval and LeetCodeHardGym across two LLMs and will be included in the final paper.

We would also like to point out that the use of memory banks is now a seemingly standard best practice that ANY iterative prompting approach should incorporate (we compare and contrast again some recent works that use memory-banks in Section-4); this is our main reason for including it in our work. While DoT-Bank shows higher performance, DoT itself surpasses prior state-of-the-art methods, underscoring the independent value of both components.

W2. We thank the reviewer for raising these concerns. First, we would like to clarify that the primary focus of our method is also on maximizing accuracy, with efficiency as an ancillary benefit. We highlighted the efficiency comparison with LATS because its high cost is tied to its inherent statistical inefficiency, caused by redundant reflections—a key issue our method directly addresses. The Diverse-Reflections module in DoT demonstrates that effective results can be achieved without resorting to complex, resource-intensive methods, offering both efficacy and efficiency.

Regarding the experimental setup, we have followed the same methodology as foundational papers in this area, such as Reflexion, Tree-of-Thoughts, and LATS. While prior works did not report statistical significance testing, we agree that it is essential to validate our results further. As a first step, please see the new table below that includes Pass@1 averaged over 3 runs for HumanEval and LeetCodeHardGym datasets using Sonnet-3.5, which we will incorporate into the final version of the paper.

Updated Results (Pass@1) with Statistical Significance (LLM: Sonnet-3.5, averaged over 3 runs)

On the claim that our gains are incremental, we respectfully disagree. Reflexion introduced LeetCodeHardGym as a benchmark for challenging tasks, and our method (DoT) achieves up to 12.5% improvement over Reflexion and LATS across various LLMs. Furthermore, on average:

1. DoT solves 7 more problems on HumanEval, 18 on MBPP, and 2 on LeetCodeHardGym compared to prior best methods.
2. DoT-Bank solves 14 more problems on HumanEval, 36 on MBPP, and 4 on LeetCodeHardGym compared to previous best methods.

These improvements are not tuned for specific LLMs or datasets, reinforcing that the gains are robust and not attributable to random variation. The consistent performance across datasets and LLMs provides additional evidence of the generality and significance of our approach.

I appreciate the author's effort to address my concerns, particularly their detailed response and inclusion of new experiments. However, two main concerns remain.

First, regarding its practical usability: while the authors demonstrate improvements over foundational algorithms like LATS and reflexion in both accuracy and efficiency, to substantiate claims of SOTA performance in Agent framework design in coding tasks, which are the main focus of this paper, they should include comparisons with current SOTAs in the field [1, 2]. Additionally, the discussion about time efficiency would benefit from addressing computational overhead related to embedding vectors and similarity calculations. Given the significant engineering complexity of this work, at least some short sample anonymized code release would also be very helpful for reproducing the work.

Second, concerning the paper's core contribution about diverse thought helping LLM reasoning: this idea, while valuable, has been explored in several recent works [3, 4], and the current paper lacks either theoretical foundations or social science perspectives to differentiate its contribution from existing literature.

Nevertheless, I commend the authors' efforts in addressing my initial concerns, their comprehensive experiments and ablation studies, and the clear writing style. Given these considerations, I am willing to raise my score to 6, though I still remain neutral on the accept/reject decision and defer to other reviewers and ACs.

References: [1] Zhong, L., Wang, Z., & Shang, J. (2024). Debug like a Human: A Large Language Model Debugger via Verifying Runtime Execution Step by Step. Findings of ACL 2024, 851-870.

[2] Huang, D., Zhang, J. M., Luck, M., Bu, Q., Qing, Y., & Cui, H. (2024). AgentCoder: Multi-Agent-based Code Generation with Iterative Testing and Optimisation. arXiv preprint arXiv:2312.13010.

[3] Hayati et al. (2024). How Far Can We Extract Diverse Perspectives from Large Language Models? EMNLP 2024.

[4] Muscato, B., Mala, C. S., Manerba, M. M., Gezici, G., & Giannotti, F. (2024). An Overview of Recent Approaches to Enable Diversity in Large Language Models through Aligning with Human Perspectives. Proceedings of the 3rd Workshop on Perspectivist Approaches to NLP, 49-55.

We sincerely thank the reviewer for their constructive feedback and appreciate their willingness to raise the score. Below, we address the two main concerns raised:

Comparisons with Recent SOTA and Practical Usability: We appreciate the reviewer pointing us to recent SOTA works in agentic frameworks for coding tasks. In response, we conducted additional experiments comparing DoT variants against LDB [1], an accepted and well-established work, on the HumanEval dataset using GPT-4o and GPT-4o-mini. We report Pass@1 results (averaged over three runs with standard deviations) below. Notably, DoT variants demonstrate superior performance over LDB. We hypothesize that integrating DoT-generated seed programs into LDB’s framework may further enhance its performance. This highlights the complementary nature of our contributions.

Regarding computational overhead, as mentioned in our prior response, embedding similarity computations are negligible for datasets like HumanEval and MBPP, especially when embeddings are generated using efficient APIs. We will include a detailed discussion of this in the final version.

Additionally, we would like to clarify that all code and log files have already been released as part of the supplementary material in the initial submission to facilitate reproducibility.

Contribution on Diverse Reflections Enhancing Reasoning: We note that [3] was recently accepted shortly before the ICLR25 abstract deadline, and we appreciate the opportunity to incorporate its insights in our updated PDF. We acknowledge that [3, 4] have explored one-shot sampling to produce semantically diverse outputs. However, these works do not specifically investigate how diversity aids reasoning—a key focus and contribution of our study.

Our work identifies principled shortcomings in prior iterative frameworks like Reflexion and LATS and demonstrates how diversity in reflections can address these issues. This contribution is novel and, to the best of our knowledge, represents the first exploration of diversity-enhanced reasoning in this context. While [3, 4] align with our findings, they provide orthogonal insights rather than overlapping contributions. Also, our updated PDF and previous response acknowledges [3].

We hope these clarifications address the reviewer’s concerns and reinforce the novelty and practical value of our work. We appreciate the detailed review and look forward to further feedback.

We thank the reviewer for their constructive feedback and are glad that they found our work to be well-motivated and well-justified. We are encouraged by the recognition of our carefully chosen experiments and the clarity and consistency of our reported gains in performance.

W1: We will revise the writing to clarify how previous trajectories are retrieved and to better explain our ablative study in Table 9. Briefly, Table 9 examines the impact of injecting random in-context examples versus cosine-similarity-based examples. In phase 1, successful trajectories and the embedding of the problem's docstring are added to the memory bank to facilitate retrieval. In phase 2, when re-attempting failed problems, we generate an embedding for the docstring and retrieve the k most similar trajectories from the memory-bank to inject as in-context examples. The results indicate that using similar-style examples boosts performance over random ones, aligning with recent findings in [1].

W2: We will revise the plots to place both embeddings in the same space, as suggested, to improve clarity. Additionally, we introduce a new metric to quantify the diversity of generated reflections. This metric calculates the average pairwise cosine similarity of self-reflections for each problem within a dataset; we report both the mean and standard deviation. We use the ‘all-Mini-LM-v6’ model from SentenceTransformers to generate embeddings for these reflections. Diversity scores are presented for HumanEval and LeetCodeHardGym across two different LLMs, with lower similarity scores indicating greater diversity.

W3: We agree with the reviewer. We will reformat and reposition the tables to enhance readability and clarity.

Q1: Tree-of-Thoughts and LATS use LLMs as evaluators, where the LLM is prompted to generate a numeric score, as noted by the reviewer.

Q2: Yes, the K most similar trajectories are injected as in-context examples specifically for generating function implementations, not reflections. As stated in the paper, the task-agnostic memory bank aims to enhance generation quality.

References

[1] Metacognitive Capabilities of LLMs: An Exploration in Mathematical Problem Solving, Didolkar. et. al, NeurIPS 2024

We thank the reviewer for their valuable feedback.

W1: We disagree with the reviewer’s comment regarding the practicality of reusing insights from previously solved tasks. Our empirical results underscore the practical value of DoT-Bank, showing a consistent 2-5% performance improvement over the standard DoT approach—validating that leveraging past insights enhances task-solving effectiveness.

A recent concurrent study [1] demonstrates that Large Language Models (LLMs) can effectively use metacognitive abilities to leverage previous insights when approaching new tasks, leading to substantial performance improvements. Specifically, they show that curating in-context examples based on skill-exemplars—i.e., insights from related tasks—enhances performance. However, their approach relies on a predefined set of exemplars, requiring additional training data, which may limit adaptability across varied tasks.

Our proposed task-agnostic memory bank variant, "DoT-Bank," corroborates this observation but innovates by eliminating the need for explicit training data. Instead, DoT-Bank dynamically builds a memory bank during problem-solving, effectively capturing useful knowledge from previously solved tasks. This enables DoT-Bank to leverage cross-task insights without task-specific exemplars.

W2: As stated in Line 300, we use the recommended hyperparameters suggested by the authors of the respective baselines. Specifically, for Reflexion, we set max-iterations k=3 for HumanEval and MBPP, and k=5 for LeetCodeHardGym. For LATS, we use k=8 across all three datasets. For DoT and DoT-Bank, we set k=3 for HumanEval and MBPP, and k=5 for LeetCodeHardGym.

To further highlight DoT's effectiveness, we vary k in Reflexion and report both Pass@1 and associated costs on the HumanEval dataset with Sonnet-3.5. The results indicate that even with k=10, Reflexion lags behind DoT and DoT-Bank, which achieve Pass@1 scores of 91.46 ($1.14)** and **93.90 ($1.67), respectively—demonstrating that DoT variants are both more effective and cost-efficient.

If the reviewer is referring to sampling k-reflections per iteration, this approach is already implemented in LATS, which additionally uses an MCTS mechanism to select a potential solution (see Section 5.2 paragraph 2 in LATS). Our empirical results show that DoT outperforms LATS, and in the few cases of similar performance, DoT remains more cost-effective.

W3: We clarify that by “cost”, we refer solely to the dollar cost incurred by running the LLM APIs. The memory bank is implemented as a hash map, which does not contribute to this cost. It appears there may have been a misunderstanding regarding our algorithm. The memory bank's size increases with each new solved task (Algorithm 1, Lines 17 and 35). DoT-Bank includes both phases of the algorithm: phase 1, which builds the memory bank and tags failed problems on visible or synthetic test cases, and phase 2, which reattempts these tagged problems using the memory bank. For all DoT-Bank experiments, both phases are run. This is why, in all relevant tables, DoT-Bank's cost is higher than DoT’s—reflecting the additional cost of reattempting failed problems with extra context tokens and the API cost for generating embeddings.

The memory bank is only applied to failed tasks, as shown on Line 32 of Algorithm 1.

W4: We use the same set of models for HumanEval and MBPP experiments. For the LeetCodeHardGym benchmark, consistent with Reflexion, we utilize powerful LLMs as base models, specifically GPT-o1 and Sonnet-3.5 in the main paper, and GPT-4o/4o-mini in Table 11 of Appendix.

Table 5 highlights a bug in the LATS implementation that led to inflated performance results on some datasets. To illustrate this, we use the same setting from LATS with synthetic test cases. All baseline results in Table 5 are taken directly from the LATS paper, alongside our corrected results for LATS and DoT.

Lastly, we emphasize that, unlike previous works, our study evaluates a broad range of LLMs, demonstrating consistent performance gains across different model types.

W5: For ToT + DoT, we use one-shot-sampling along with the prompt mentioned in Figure 8. A recent concurrent study [2] supports our approach, demonstrating that one-shot sampling can produce semantically diverse outputs.

We updated our paper to make these points clear in addition to adding new experiments and discussions.

References:

[1] Metacognitive Capabilities of LLMs: An Exploration in Mathematical Problem Solving, Didolkar. et. al, NeurIPS 2024

[2] How Far Can We Extract Diverse Perspectives from Large Language Models? Hayati. et. al., EMNLP 2024

Thank you again for your review. As the discussion period is concluding soon, we would like to follow up to ensure we have sufficiently addressed your concerns and to kindly ask if you might consider reassessing your evaluation.

To summarize our updates:

1. We added a connecting paragraph in Section I, citing a relevant recently accepted work to strengthen support for our memory-bank (insights from previous tasks are helpful) and improve the narrative coherence.

2. We included a new study in Appendix A.2.5 to show that naively increasing total generated reflections in Reflexion has minimal impact on performance. Appendix A.2.1 shows that our reported gains are statistically significant. We hope that these new experiments and clarifications make our contributions clear.

3. We additionally included a diversity metric in updated Section 3.3.1 to show that DoT can reduce redundant reflections compared to baselines.

We hope these additions clarify our contributions and address your concerns. We would be happy to answer any follow-up questions and address any remaining concerns.

Dear Reviewer RxnW,

We hope our response has adequately addressed your concerns. As the discussion period nears its conclusion, we kindly ask if you have any remaining questions or if you could reassess our paper based on the clarifications provided.

Thank you for your time and consideration.

We thank the reviewer for their valuable feedback.

W1: We clarify that our method does not claim to guarantee diverse reflections but instead aims to promote them (L::71-73). To quantify the diversity, we calculate the average pairwise cosine similarity across generated self-reflections for each problem within a dataset, reporting both the mean and standard deviation. We use the ‘all-Mini-LM-v6’ model from SentenceTransformers [1] to generate embeddings for these reflections. This metric is presented for HumanEval and LeetCodeHardGym across two different LLMs. A lower similarity score indicates greater diversity.

To clarify further, in addition to using an explicit prompt, we also incorporate one-shot sampling to enhance the diversity of the generated self-reflections. Moreover, the prompt is supplemented by feedback from the environment, as well as context from previous implementations and self-reflections. A recent concurrent study [2] supports our approach, demonstrating that one-shot sampling can produce semantically diverse outputs. In the final version, we will include this diversity metric across all relevant tables and provide a citation for the referenced work. The tables below demonstrate that both DoT and DoT-Bank exhibit lower similarity scores compared to baseline methods, further supporting the effectiveness of our approach in reducing redundancy within reflections. Figure-4 corroborates this observation.

W2 (Concerns on Contributions): We believe that we are the first to identify repetition in reflections as a limitation in self-reflection-based frameworks—a key contribution of our work. In Section 2.1, we clarify that DoT builds on Reflexion, but we observe that substituting M_sr with M_dsr to enhance diversity yields consistent improvements over both Reflexion and LATS across various base LLMs. Additionally, as shown in Algorithm 1 (lines 20-22), DoT differs subtly but critically: rather than generating a single reflection per iteration like Reflexion, DoT produces ‘k’ diverse reflections and selects the most promising one, effectively emulating a greedy BFS approach.

Our Task Agnostic Memory Bank further complements DoT, enhancing generation quality. Section 4 provides a comparison with recent memory-based methods; if possible, we would appreciate specific references to any memory-related works the reviewer has in mind, as this could help further contextualize our contributions. We have also included code and logs of generated implementations and reflections in the supplementary files to allow for a qualitative examination of DoT's reflections.

DoT variants address the identified shortcomings and consistently improve performance across various programming benchmarks. Notably, on the challenging LeetCodeHardGym dataset (introduced in Reflexion), where previous methods show minimal improvements over base LLMs, DoT variants achieve a significant performance boost across LLMs (GPT-o1, Sonnet-3.5, GPT-4o), with gains of up to 12.5%.

Q1. Thank you for pointing this out. The missing reference in line 700 corresponds to Figure 7.

References

[1] https://sbert.net/docs/sentence_transformer/pretrained_models.html

[2] How Far Can We Extract Diverse Perspectives from Large Language Models? Hayati. et. al., EMNLP 2024

Thank you again for your review. As the discussion period is concluding soon, we would like to follow up to ensure we have sufficiently addressed your concerns and to kindly ask if you might consider reassessing your evaluation.

To summarize our updates:

1. Updated Section 3.3.1. We incorporated a diversity metric and referenced recent related work that supports our observation that one-shot sampling can produce semantically distinct outputs.

2. Contributions and Analysis. One of the key contributions of our work is the well motivated analysis to identify repeated reflections as shortcoming and using diverse reflections to alleviate it. We included a new study in Appendix A.2.5 to show that naively increasing total generated reflections in Reflexion has minimal impact on performance. Appendix A.2.1 shows that our reported gains are statistically significant. We hope that these new experiments and clarifications make our contributions clear.

We hope these additions clarify our contributions and address your concerns. We would be happy to answer any follow-up questions and address any remaining concerns.

Dear Reviewer 7Wju,

We hope our response has adequately addressed your concerns. As the discussion period nears its conclusion, we kindly ask if you have any remaining questions or if you could reassess our paper based on the clarifications provided.

Thank you for your time and consideration.