The Entity-Deduction Arena: A playground for probing the conversational reasoning and planning capabilities of LLMs

We are grateful to the reviewer for their valuable feedback and the insights provided. Below, we have addressed the questions and concerns raised:

Entity-deducing game is one small problem: We completely agree that the benchmark we used does not fully reflect the reasoning ability of the LLM. Nevertheless we thinkg this problems adds to the well needed suite of test cases for measuring these characteristics of LLMs. We will revise the paper to make this point clearer. Please refer to our general response.

Experimental Scale: We thank the reviewer for the suggestion. Please refer to the general questions for more discussion.

We thank the reviewer for providing us with constructive feedback and supportive comments! We address the questions in below:

Numerical or spatial reasoning: Please also refer to our general response. We acknowledge that this paper does not address numerical or spatial reasoning, and that was intentional. We will make it clearer that we are only evaluating specific capacities of LLMs involving deductive reasoning, state tracking, fact recalling and strategical planning, under multi-turn information querying scenarios. We really like the suggestion to expand our evaluation to include more realistic tasks (e.g., spatial planning, numerical reasoning, tool-use, and task-completion chatbot) beyond text-only objectives. To achieve that, we believe that a controlled testing environment with clear metrics and evaluation would be essential as a first step towards generalizing to diverse scenarios and prompt requirements.

Mitigation of undesirable behavior: In our experiments, we have observed that behavior cloning (BC) training can effectively mitigate undesirable behavior and generation artifact we showed in Table 9, such as repetition. We think that the model's confidence level might play a crucial role in this mitigation. When the model is uncertain about which action to take (we can probe the model by asking its current prediction and confidence level), we have noticed a tendency for it to exhibit degeneration. By incorporating reasoning and planning capabilities learned from stronger models into weaker models through BC, we can demonstrate good trajectory and enable the model to follow a more effective strategy in generating questions. Generally, we consider these issues to be fundamental to autoregressive training via teacher forcing and believe that addressing them is essential for the effective learning and adaptability of LLMs. As far as we know, mitigating these issues for LM remains an open question for further exploration [2, 3].

[2] Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In ICLR, 2019.

[3] Jin Xu, Xiaojiang Liu, Jianhao Yan, Deng Cai, Huayang Li, and Jian Li. Learning to break the loop: Analyzing and mitigating repetitions for neural text generation. In NeurIPS, 2022.

We thank the reviewer for the detailed and insightful feedback. In response, we have addressed the comments as follows:

Over-anthropomorphizing: We appreciate and agree with the reviewer's viewpoint and will tweak the language carefully to avoid the accidental over-anthropomorphization in our writing style. Note that we were not suggesting that "intelligent behavior" is human-like or human-level but we can see in retrospect how our presentation might be construed that way -- a careful review and rephrasing of parts of the paper should alleviate this.

Evaluating the Judge: We acknowledge the need for better quantification of the errors made by Judge. Since obtaining the ground truth is challenging, we decided to conduct a human evaluation of Judge (GPT3.5-turbo-0613) responses. This evaluation involved non-paid volunteers from our institute who assessed the correctness of Judge's responses to questions asked given the oracle entity or celebrity name. Due to time constraints during the rebuttal, based on our current annotation progress, we were only able to evaluate 223 Judge responses on Things and 237 Judge responses on Celebrities. The human evaluators identified 7 out of 223 (3.13%) and 7 out of 237 (2.95%) responses as problematic or incorrect. A more comprehensive summary of these results will be included in our revision.

The evaluation set is small: We agree that having a larger evaluation dataset would enable more comprehensive and detailed analysis at a finer granularity. To this end, we are currently working on expanding the evaluation by incorporating 300 examples for each dataset, across all the models being compared. We will incorporate the results of this evaluation in our general response.

Human evaluation details: We have discussed some details of creating the human baseline in the Appendix D. We agreed that some contents needed to be moved to the main text, while some information needed to be supplemented. In this study, we did not hire any paid crowd-source workers. As discussed in Section 3, because the complexity of the multi-turn game setup, we developed an internal demo server (shown in figure 4) and relied solely on enthusiastic internal volunteers who were interested in our study. Following a 30-day trial period, 108 volunteers contributed to this study, and we successfully collected a total of 216 human game play sessions for Things and Celebrities.

To ensure the quality of the data, we manually inspected human game plays to filter out low quality data. The authors reviewed all human game play sessions, and removed sessions that do not complete the entity deduction task, contain irrelavant chit-chat, or contain nonsensical repeated questions, leaving only sensible game plays. In the end 201 sessions were selected as valid game play.

We appreciate the suggestion and will incorporate a thorough discussion comparing the strategies employed by humans and LLMs into the main text.

Evaluation of Human players on reasoning tasks: We are currently conducting some additional experiments and will provide supplementary results shortly.

Deterministic models: In our study, we compute statistics for each game by performing 5 repetitions for each game. We apply a sampling approach and use a temperature of 0.8 for the guesser models throughout our study. We observed that the gameplay can vary significantly, resulting in completely different outcomes. For example, the failure results presented in Table 2, 8, 12 are derived from the same model, entity, and settings.

Evaluation of Human players on reasoning tasks:

We thank the reviewer for the valuable suggestion. While we are still awaiting the full results, we would like to share our current findings below:

Note that these results are based on the evaluation set in the submission for consistency. In our next update, we plan to extend the study to include a new dataset with 300 items/names. Following the setup outlined in the paper, only the last turn for each dialog trajectory is replaced. We are currently relying on non-paid volunteers for the annotation.

The findings indicate that when GPT-4 is employed for the last step reasoning in human trajectories, there is a decrease in performance. One possible explanation for this is that while GPT-4 may excel in comprehending model-generated trajectories, it may struggle to fully interpret the planning trajectory of humans based on the observed turns. This suggests that models may be more compatible with model-generated trajectories rather than human trajectories. For instance:

Even though it was previously established that the entity can write, GPT-4 still asked if the entity is a Stapler or not.

We thank the reviewer's comments and feedback on our work. Next we address the questions and concerns:

Not enough improvement over referenced LLM: Perhaps we misunderstood the reviewer, but we feel that our methods show strong improvement over our baselines in Table 1 -- both BC trained models (V-FT* models in Table 1 -- e.g. V-FT 7B) and RL trained models (V-RLGP* models in Table 1 -- e.g. V-RLGP 7B) show more than 100% improvement over the base model (Vicuna 7B). Perhaps the reviewer is suggesting that we compare our trained models against GPT-4 ? In that case it would be unfair since the GPT-4 model is much larger than the models we trained.

Not enough diversity in the datasets and no discussion: In section 3.1 and Appendix A, we discussed that we aim to create rich and diverse datasets. In fact, our datasets are diverse. The Things dataset encompasses a wide range of categories, such as objects, animals, foods, plants, vehicles, clothing, professions, materials, instruments, places, birds, sports, buildings, furniture, celestial bodies, mythical creatures, games, body parts, beverages, weather phenomena, groups, gemstones, toys, tools, patterns, appliances, and microorganisms, among others.

The celebrities dataset meanwhile includes names from different nationalities, eras of life, and various occupations, including entertainment, sports, politics, entrepreneurship, science, philanthropy, writing, music, military, engineering, and more. We used 980 entities for the Things dataset and 133 entities for celebrities. Additionally, we are in the process of collecting and evaluating 300 additional examples for each dataset to further expand its size. The results of this effort will be shared in our general response.

No repeated experiment: We apologize if we didn't emphasize this enough in the paper, but we did in fact repeat our experiments. In Table 1, our main results were obtained by conducting 5 repetitions for each gameplay, which provided the standard deviation as shown in the paper. This was further discussed in Section 4.

Why linearity in reward design: It is commonplace in the RL community to use rewards based on heuristics that assign numeric values commensurate with desriable properties, and then combine these scores linearly. For instance, similar heuristics have been used in controlling the agent's movements [4] and in task-oriented chatbots [5]. In this paper, the score we proposed combines a reward for a good game outcome with a penalty for taking longer to achieve the right answer and 0.02 is just a hyperparameter that balances these two properties. With a value of 0.02 the model is still rewarded 0.7 for answering a question correctly after the full 20 steps. Values greater than 0.02 would reduce the rewards for games that take longer, while smaller values would penalize the model less for taking a longer time. In our early exploratory experiments we found 0.02 to lead to models with higher final success rate, compared to other values. Admittedly, reward selection is an art in RL, rather than a well understood science and it is possible that there are better choices out there, both in the heuristics chosen, and how their values are computed and combined -- we would be happy to add such qualifications to the paper.

Lack path planning analysis: In fact, we provide a comprehensive qualitative analysis of what constitutes a good or bad path for the game. In Table 2 and Tables 8, 9, and 12, we have highlighted how the models are different in their planning ability and illustrated and explained the characteristics of good "path planning" and identify the common failure modes that lead to inadequate planning. The criteria for a good path indeed depends on the domain and may be hard to generalize at this stage of our understanding of LLMs. However, we discussed some general observations. In Table 3 and 13, we present the criteria for successful planning, which include 1) prioritizing high-level questions before addressing specific details and enumerations, 2)being aware of the current state and asking questions to effectively bi-partition the search space, and 3) being able to recognize when the current path is incorrect and occasionally backtracking to consider previously overlooked options. All of these aspects are discussed in Section 4 of our paper, and we will make them clearer.

[4] Jens, et al. "Reinforcement learning in robotics: A survey." The International Journal of Robotics Research 32.11 (2013)

[5] Bhuwan, et al. "Towards end-to-end reinforcement learning of dialogue agents for information access." ACL (2016).

Dataset size is not sufficient: We acknowledge this limitation, and have since expanded our dataset size from 30 things and 30 celebrities to 300 each, containing 600 entities in total that are not overlapping with the training datasets. Specifically, while expanding things dataset, entities were chosen from the various categories including objects, animals, foods, plants, vehicles, clothing, professions, materials, instruments, places, birds, sports, buildings, furniture, celestial bodies, mythical creatures, games, body parts, beverages, weather phenomena, groups, gemstones, toys, tools, patterns, appliances, and microorganisms. For celebrities datasets, the additional datasets cover names from different nationalities, eras of life, and various occupations. The updated experiment results can be found below:

Overall, it is observed that the performance on the new evaluation datasets decreased for all methods, which is likely attributed to the increased difficulty of the evaluation set. The relative comparison between different methods largely remains unchanged, except for Claude-1 which slightly outperforms Claude-2. Owing to time limitations and the larger number of evaluation items, we were only able to conduct one repetition for each entity or name. We are currently in the process of collecting complete results with five repetitions and other models in the paper. Once they are available, we will revise the paper accordingly.