From Passive to Active Reasoning: Can Large Language Models Ask the Right Questions under Incomplete Information?

Thanks for the valuable feedback. To solve these concerns and questions, we provide point-to-point responses as follows.

Q1. The reliability and necessity of using LLM as a judge, and evaluating the judge on Turtlebenchmark.

Reply: Here, we (1) explain why we use LLM judges instead of human judges, (2) clarify our choice of TurtleBenchmark for evaluation, and (3) present further evidence supporting the reliability of LLM judges on AR-Bench.

(1) We use LLMs as judges because model-based evaluation is essential for active reasoning. Human evaluation in multi-turn interactions is prohibitively expensive, so we employ LLMs as a cost-effective alternative. Recent studies [1, 2] support the effectiveness of LLMs as judges, showing that they can reliably simulate human judgment. This approach enables large-scale evaluation, which is crucial for studying active reasoning.

(2) We use TurtleBenchmark to evaluate LLM judges because it provides human-verified annotations from real-world games. TurtleBenchmark includes 4,448 high-quality samples, making it a reliable dataset for testing LLM-as-a-judge capabilities. However, it does not support process-level evaluation, such as assessing the quality of the questions asked by LLM players. For this reason, we use AR-Bench to evaluate active reasoning abilities and TurtleBenchmark to assess the accuracy of LLMs as judges.

(3) We run additional experiments with AR-Bench to further verify the reliability of LLM judges. Here, we collect 200 questions from the SP tasks of AR-Bench and manually annotate their answers. Then, we evaluate various LLM judges on this collected dataset. The results, shown in the table below, align closely with findings from TurtleBenchmark. Notably, Llama3.1-405B achieved 96% accuracy, demonstrating reliable judgment on AR-Bench.

[1] LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods. In arXiv, 2024.

[2] Let’s verify step by step. In ICLR, 2024.

Q2. The potential of designing a simpler method to provide information to the player.

Reply: We would like to (1) discuss the potential of using smaller LLMs as the judges and (2) explore the possibility of using rule-based functions as judges in certain scenarios.

(1) Smaller LLMs can serve effectively as judges. As shown in Q1, smaller and more efficient models like QwQ-32B achieve performance comparable to larger models such as Llama3.1-405B, while being significantly more cost-effective. Besides, based on the price in TogetherAI, the API cost of Llama3.1-405B is 3.50 USD per million tokens, whereas QwQ-32B costs only 1.20 USD per million tokens. These results support the feasibility of using advanced smaller LLMs as judges.

(2) Rule-based functions can effectively act as judges when the action space is limited. For example, in tasks like GN, where the possible actions are predefined (e.g., 5,040 unique four-digit numbers), rule-based judgment is feasible. Although this constrains the player’s actions to a fixed set, it eliminates the need for an LLM-based judge. In contrast, tasks like SP and DC encourage open-ended question generation, making it necessary to use an LLM judge to handle the broader, more flexible action space. Overall, rule-based evaluation is a promising direction for assessing active reasoning tasks. We believe it’s worth further exploration, especially to address the challenges posed by open-ended scenarios.

Q3. Discuss the future work on new training strategies.

Reply: We would like to answer this question from the following two perspectives.

(1) Data perspective. It is feasible to create small-scale, high-quality datasets for fine-tuning large language models in active reasoning tasks. As with s1 and LIMO, these datasets should capture the detailed thinking and interaction process involved in active reasoning—revealing how to ask effective questions and ultimately make a final decision. Because current LLMs often struggle with generating high-quality questions, it may be necessary to curate such data through human annotation, enabling models to learn directly from human question-asking strategies.

(2) Algorithm perspective. We can leverage reinforcement learning techniques with outcome-based rewards, drawing inspiration from methods like PPO and GRPO. Instead of assigning a reward after every question, the model receives a significant reward only if it arrives at the correct solution, thereby eliminating the need to manually label the quality of each individual question. This reduces annotation costs while naturally promoting planning, exploration, and self-reflection in the model’s questioning process.

We will include the above clarifications and discussions in the revision. Any further comments and discussions are welcome! Thank you!

Thanks for the valuable feedback. To solve these concerns and questions, we provide point-to-point responses as follows.

Q1. The selection of language models to compare.

Reply: We provide a two-fold answer as follows.

(1) Considering the huge expense of computing and API calls, we evaluate the currently representative LLMs on AR-Bench. Specifically, we select GPT-4o and GPT-4o-mini as representatives of closed-source LLMs; and Llama3.1 8B, 70B, and 405B as representatives of open-source LLMs. These models are widely used and compared in current leaderboards and arenas, which include both reasoning and non-reasoning models. In our experiments, we guarantee fair evaluations and comparisons by providing the same information to different models under evaluation.

(2) We conduct extra experiments with more language models and show their general deficiency in active reasoning tasks. We involve Qwen2.5 (3B, 7B) - another representative model family, and QwQ-32B - a powerful reasoning model. The evaluation result shown below is consistent with our observation. This highlights the widespread challenges that current LLMs face in active reasoning tasks.

Q2. Ablation study with various model configurations of temperatures and topk.

Reply: In our experiments, we used consistent hyperparameter settings with temperature = 0.7 and top-p = 0.7. We did not modify the top-k parameter, as it is not configurable in the OpenAI API. Further, we present extra ablation studies and show that varying temperature and top-p values resulted in only marginal differences in performance across active reasoning tasks. As shown in the tables below, extreme values of temperature and top-p (either too high or too low) lead to suboptimal performance in SP and DC tasks, while having minimal impact on GN.

Q3. The human intervention in generating the dataset.

Reply: We answer this question in three folds:

(1) We used a rigorous multi-stage process to ensure the reliability and logical consistency of each AR-Bench sample. Specifically:

• For DC, the core logic remains intact. Each puzzle has a uniquely identifiable murderer based on motive, opportunity, and weapon access. The synthetic process only adds narrative details without affecting the deductive structure.
• For SP, which is based on lateral thinking, any logically consistent explanation is valid. We start by sampling a core counterfactual fact and use tree-based expansion to generate explanatory questions and corresponding facts (see Figure 4).
• For GN, the task is simple and the synthetic process is reliable, as it only involves generating four-digit numbers with unique digits.

(2) We conducted manual checking of the puzzles and ground truths in AR-Bench to ensure they are reliable, logically consistent, and solvable through reasoning. Specifically:

• For DC, we verify that the murderer has all key traits (motive, opportunity, weapon access) and that innocent suspects lack at least one.
• For SP, we check that the provided answer logically explains the scenario and maintains internal consistency.
• For GN, we confirm that each number consists of four unique digits.

Besides, as introduced in Appendix A, we provided the source files of AR-Bench in an anonymous repository. The reviewers can directly check the quality of data points in the AR-Bench.

(3) In addition, synthetic data offers several advantages. Synthetic data reduces overlap with existing datasets for mitigating data leakage, scales efficiently without the cost of human-annotated examples, and is easily adaptable for creating new active reasoning tasks.

Q4. The human evaluation of the benchmark.

Reply: Considering the unaffordable expense of large-scale human evaluation, we conduct an extra demo-level human evaluation with two undergraduate students on AR-Bench to show human performance (please see the table below). Despite the inherent challenges in human evaluation, the results still reveal a substantial performance gap between humans and LLMs, underscoring the critical need to enhance active reasoning capabilities in current language models.

We will include the above clarifications and discussions in the revision. Any further comments and discussions are welcome! Thank you!

Thanks for the valuable feedback. To solve these concerns and questions, we provide point-to-point responses as follows.

Q1. The benchmark’s novelty and consistency with real-world scenarios.

Reply: We answer this question from three aspects.

(1) Reality. Real-world active reasoning can be messy, but benchmarks initially need simpler, controlled tasks to isolate specific capabilities, i.e., effectively asking questions to obtain missing information. Synthetic puzzles ensure consistency and clear metrics, while the yes/no feedback is more structured than real interactions. We plan to add more nuanced “deceitful” NPCs over time.

(2) Novelty. Although our puzzles resemble known games, combining them into a single benchmark systematically tests active reasoning, which is an overlooked dimension. We provide 6,000+ puzzles, curated prompts, automated feedback, and extensive experiments, showing current LLMs struggle primarily due to inadequate questioning. Pinpointing how and why it fails is our key contribution.

(3) Influence. We will release all data and code publicly, enabling easy reproduction and further improvement. AR-Bench fills a gap in current reasoning benchmarks, which largely assume complete information is available. Our controlled environment highlights specific failings and can evolve to include richer, more realistic forms of active reasoning.

Q2. The error patterns of LLMs on AR-Bench.

Reply: We carefully check the 600 running dialogs of Llama3.1-8B and GPT-4o on AR-Bench and categorize typical errors in each task.

In DC:

• Timeline Misinterpretation (TM): Models confuse the timeline of the murderer and ask for information of an irrelevant time.
• Evidence Overlooked (EO): Models ask for details and overlook the key evidence needed to find the murderer.

In SP:

• Evidence Overlooked (EO): Models jump to conclusions without uncovering supporting evidence.
• Unsupported Assumptions (UA): Models fabricate fake details in the conclusion.

In GN:

• Feedback Misunderstanding (FM): Models misinterpret feedback, failing to keep correct digits or eliminate wrong ones.
• Incomplete Testing (IT): After identifying a correct digit, models fail to determine its correct position.

The following tables show error statistics for Llama3.1-8B and GPT-4o.

• In DC, GPT-4o overlooks less key evidence but struggles more with timeline issues.
• In SP, Llama3.1-8B fabricates details more often, while GPT-4o tends to miss details.
• In GN, Llama3.1-8B makes more errors overall, though GPT-4o also has high error rates.

Q3. Explain the technical details and empirical results of SFT.

Reply: We train the model separately for each task. Our analysis of the SFT results reveals the following:

SFT tends to memorize training data rather than develop active reasoning skills. This is especially problematic in the GN task, where memorizing number patterns doesn’t teach the logical deduction needed for accurate guessing, resulting in 0% accuracy on unseen test cases.

Moreover, the training data includes only the output questions, without showing the underlying reasoning behind them. This lack of reasoning traces limits learning, especially in symbolic tasks like GN. In contrast, DC data (with its free-form questions) better supports the development of active reasoning, leading to improved performance.

Q4. Attempt related work Proactive CoT and UoT on AR-Bench.

Reply: We adapt these two methods to AR-Bench:

• Proactive CoT: Pre-defined task-specific strategies and actions based on human reasoning patterns.
• UoT: In DC, we sample 3 question branches, simulate 1 turn, and estimate uncertainty reduction. In SP, we initialize the potential answers, sample 3 questions/turn, and estimate uncertainty reduction. In GN, we sample 3 guess branches, simulate 1 turn, and use the model to estimate uncertainty reduction.

The results below show that both methods cannot solve AR-Bench well. This demonstrates the necessity of designing new methods (please see Q3 to reviewer Briw).

Q5. The robustness of the LLM judges against irrelevant questions.

Reply: We instruct judge models to withhold useful information when questions are irrelevant. In SP, they respond with “unknown”. In CD, while suspects may reply, they avoid revealing useful information. We illustrate this with running examples of irrelevant questions.

Q6. Human evaluation on AR-Bench.

Reply: Due to the high cost of large-scale human evaluation, we conducted a small demo with two undergraduate students on AR-Bench to show human performance. Please see our response to Q4 for Reviewer Vrs4.