Rethinking Chain-of-Thought from the Perspective of Self-Training

We sincerely thank the reviewer for the valuable feedback and encouraging comments. We are motivated by the suggestions and have addressed each concern in detail as follows:

Q1. There is a lack of discussion on negative results. While the paper primarily demonstrates the advantages of the method, it does not provide an explanation for the model's poor performance on certain datasets, such as why the improvement in the "commonsense" task is limited.

A1. We acknowledge that the performance gain is relatively limited on commonsense tasks. This is mainly because such tasks rely more on pretrained world knowledge and co-occurrence patterns, rather than explicit logical steps. In contrast, arithmetic tasks better align with our iterative reasoning framework. For open-ended tasks, additional reasoning may introduce noise rather than benefit. We will explore better adaptations to such tasks in future work.

Q2. The main text lacks a Related Work section, and the Task-Specific Prompt part of Framework Figure 4 does not clearly describe the specific details, making it difficult for readers to understand how to obtain the candidate prompts and select the optimal one.

A2. In the final version, we will add a “Related Work” section to cover relevant studies on Self-Training, Chain-of-Thought reasoning, and semantic entropy. We will also refine Figure 4 and clarify that the task-specific prompt module generates candidate prompts based on several example questions, evaluates them using average semantic entropy on a validation set, and selects the one with the lowest entropy as the final prompt.

Q3. Is it better to have a larger maximum number of iterations in the proposed ARI module?

A3. We have added a sensitivity analysis of the maximum number of iterations $T$, with the experimental results shown in Table. Overall, increasing the maximum number of iterations does not always lead to better performance. Taking AQuA and AddSub as examples, the model already achieves a relatively high accuracy at $T=3$, while further increasing $T$ to 4 or 5 leads to saturated or slightly fluctuating performance, along with a significant increase in computational cost. At the same time, for more challenging tasks, a larger maximum iteration count may offer a broader reasoning space and thus yield further performance improvements. Therefore, the optimal value of $T$ should be determined based on task difficulty: for simpler tasks, a smaller $T$ is more efficient; whereas for more complex tasks, allowing a larger $T$ may better support sufficient reasoning.

Q4. This paper should include an intuitive analysis of the prompts selected by the TSP, such as whether they provide stronger guidance or how they structurally differ from manually crafted prompts.

A4. Our proposed TSP module is designed to automatically generate initial reasoning prompts that better align with the task’s semantic distribution, serving as a replacement for generic prompts. While generic prompts possess a certain degree of generality, their expressions tend to be overly abstract and fail to effectively guide the model to focus on the key reasoning elements within a task. In contrast, TSP leverages actual task samples to generate multiple candidate prompts and selects the most effective one based on average semantic entropy, thereby improving the quality of initial reasoning and reducing the number of iterations. Structurally, prompts generated by TSP typically extend the generic prompt template with additional task-specific semantic cues. For example, for the AQuA dataset, TSP produces a prompt: "Let's think step by step, how to break down the mathematical operations involved in each problem and identify the key concepts to solve them accurately".

Q5. Does the number of self-consistency samples have any particular impact on the proposed method?

A5. We conducted additional experiments on four datasets to evaluate the impact of the sampling number $N$ on model performance and robustness, as shown in Table below. The results show that as $N$ increases, the overall model accuracy improves, with more pronounced gains on challenging tasks such as AMC2023 and Gaokao. Meanwhile, performance tends to saturate after $N=3$, indicating that even a relatively small $N$ can yield stable semantic entropy estimates. This balances performance with computational efficiency and demonstrates the method’s strong robustness with respect to the parameter $N$.

We sincerely thank the reviewer for the valuable feedback and encouraging comments. We are motivated by the suggestions and have addressed each concern in detail as follows:

Q1. Main experiments should compare both effectiveness and token overhead of each method.

A1. We have added token consumption to the main experiments to assess performance-cost trade-offs. Our method outperforms others while using fewer tokens than the SOTA method (e.g., Nash), especially on challenging tasks like AMC2023 and Gaokao, showing better efficiency in complex reasoning.

Q2. Lack of sufficient theoretical claims on LLM Semantic Entropy.

A2. The highly nonlinear nature of LLMs makes it difficult to model their output distribution using traditional probabilistic methods (e.g., GMM), posing challenges for establishing a rigorous theoretical framework for semantic entropy. This paper focuses on proposing a practical, scenario-driven CoT optimization framework, using semantic entropy as a heuristic metric to capture answer distribution dynamics during reasoning. While its theoretical foundation is not fully established, experiments demonstrate its effectiveness, and we will further investigate its theoretical basis in future work.

Q3. I have concerns about its adaptability and generalization ability in more scenarios.

A3. To evaluate the generalization of the semantic entropy mechanism across tasks, we added experiments on two distinct datasets: Race (English reading comprehension task) and Anli (adversarial natural language inference task), each with 300 test samples. As shown in Table, our method achieved clear performance gains on both, demonstrating strong robustness across language understanding tasks. We plan to explore its application to Vision and Cross-Modal tasks in future work.

Q4. The assumption of a mixture of Gaussians in the proof seems overly ideal.

A4. We adopt GMM as the modeling assumption based on two reasons: (1) GMM is a widely used and interpretable model that helps illustrate how pseudo-labels reduce entropy; (2) our results are a direct corollary of [1], which provides a general derivation under the sub-exponential family, of which GMM is a special case. To balance rigor and clarity, we use GMM as a heuristic example, while the conclusions also hold under broader distributions given certain conditions.

[1] Frei S,et al. self-training converts weak learners to strong learners in mixture models.

Q5. How to determine the optimal predefined threshold $\delta$?

A5. The predefined threshold $\delta$ is empirically determined based on the task type and the number of samples $N$ in self-consistency (SC). Specifically, we allow up to $k$ predictions to deviate from the majority class and compute the corresponding entropy threshold using the ratio $(N - k)/N$. For example, in the more challenging reasoning task AQuA, when $N = 4$ and $k = 1$ (i.e., allowing at most 1 out of 4 predictions to differ from the majority), the corresponding threshold is calculated as: $\delta = -\left( \frac{N-k}{N} \log \frac{N-k}{N} + \frac{k}{N} \log \frac{k}{N} \right) \approx 0.811$.

As shown in Table, setting the threshold too high may cause LLM to terminate early before identifying the correct answer, thus harming accuracy. Conversely, setting it too low may lead to excessive reasoning and the introduction of noise, which can also degrade performance. Therefore, we typically set $k$ between 0 and 2, and use it to compute semantic entropy as the decision threshold $\delta$. As $N$ increases, we appropriately raise $k$ to enhance the robustness of the decision-making process.

Q6. Does the pipeline generalize across tasks, or must the optimized CoT be re-tuned for each one?

A6. Our framework includes a reasoning pipeline composed of the TSP and ARI modules. While ARI is task-agnostic and requires no adaptation, TSP involves optimization on a single source task. Once optimized, the entire pipeline can be directly applied to other tasks of a similar nature without re-tuning. This demonstrates the generalizability of our pipeline across tasks.

We sincerely thank the reviewer for the valuable feedback and encouraging comments. We are motivated by the suggestions and have addressed each concern in detail as follows:

Q1. Although the authors propose a method for automatically searching for the "optimal prompt," they do not provide specific examples in the experiments, nor do they analyze why it outperforms general prompts (e.g., "let's think step by step").

A1. Our proposed TSP aims to automatically generate initial reasoning prompts that better align with the specific task’s semantic distribution, serving as a replacement for generic prompts. Although generic prompts exhibit a certain degree of adaptability across most tasks, their expressions are relatively generalized and often fail to sufficiently guide LLMs to focus on the critical reasoning paths required by the task. The TSP module leverages real task samples to guide the LLM in generating multiple candidate prompts, then selects the optimal one based on the evaluation metric of average semantic entropy. This enhances the quality of initial reasoning and reduces the number of iterations. Below are examples of optimal prompts automatically searched on different datasets, showing how they further integrate task-specific features on top of the generic template:

• AQuA: "Let's think step by step, how to break down the mathematical operations involved in each problem and identify the key concepts to solve them accurately."
• AddSub: "Let's think step by step, how to efficiently solve these word problems involving basic arithmetic operations and simple logic."

Q2. Some technical details lack contextual background. For instance, the "Jaccard Index" is mentioned directly in the method section, but its role or the rationale for choosing this metric is not explained.

A2. We introduce the Jaccard Index in the method section to measure the lexical-level similarity between reasoning results of adjacent iterations, aiming to determine whether the current iteration has generated sufficiently diverse reasoning paths, thereby avoiding repetitive or redundant reasoning by the model. The Jaccard Index is chosen for its simplicity and efficiency in evaluating the overlap between two sets, making it particularly suitable for quickly assessing word-level differences between texts and effectively capturing trends in reasoning diversity. We will supplement the relevant background explanation in the paper to enhance readability and completeness.

Q3. There is no sensitivity analysis of the maximum number of iterations.

A3. We have added a sensitivity analysis of the maximum number of iterations $T$, with the experimental results shown in Table. It can be observed that the model achieves a significant accuracy improvement at $T=3$, and the performance tends to saturate or even fluctuate slightly after $T=4$, indicating that a relatively small number of iterations is sufficient to achieve near-optimal reasoning performance. Taking the AQuA and AddSub datasets as examples, the accuracy improvement slows down notably after $T=3$, suggesting limited gains from further iterations. Combined with our proposed semantic entropy-based early stopping mechanism, the system can dynamically decide whether to continue reasoning for each sample, effectively reducing unnecessary computational overhead while maintaining performance.

Q4. The symbol definitions in the formulas are not clear.

A4. Thank you for the comment. We have carefully double-checked the paper and revised the notations. In the final version, we will ensure that all symbols are clearly defined at first use.

Q5. Why does the proposed method perform better on arithmetic datasets than on commonsense datasets?

A5. We believe this difference primarily stems from the varying degrees to which different task types rely on the capabilities of language models. Arithmetic datasets emphasize strict logical steps and step-by-step calculation processes, which closely align with our method’s design that involves multi-turn reasoning and explicit intermediate steps. The model can iteratively approach the correct answer, thus benefiting significantly. In contrast, commonsense tasks rely more on the world knowledge and language co-occurrence patterns acquired during pretraining. These tasks have a more open reasoning space, more diverse answer forms, and often lack a clear logical chain. As a result, multi-turn reasoning offers limited gains for such tasks and may even introduce unnecessary redundant information that affects the final judgment.

We sincerely thank the reviewer for the valuable feedback and encouraging comments. We are motivated by the suggestions and have addressed each concern in detail as follows:

Q1. The paper lacks sensitivity analysis for the sampling number $N$.

A1. We evaluated the impact of $N$ on four datasets. Accuracy improves with larger $N$, especially on harder tasks, but saturates after $N=3$, suggesting small $N$ is sufficient for stable semantic entropy estimation.

Q2. TSP incurs extra cost. Can the authors empirically show its advantage in both performance and efficiency over traditional prompt selection?

A2. We compared prompt selection methods, including traditional ensemble prompting (generates multiple prompts and aggregates outputs) and iterative search (automatically generates and filters prompts using Monte Carlo search). These were compared with the TSP module based on semantic entropy. TSP achieves the highest accuracy across all datasets while significantly reducing total token consumption (prompt search $S_{\text{Token}}$ + inference $T_{\text{Token}}$). Compared to the search method, TSP balances performance and efficiency better.

Q3. How can the entropy principle from self-training, which involves parameter updates, be applied to CoT that only modifies inputs?

A3. We attempt to draw an analogy between the principle of self-training and CoT. Although they differ in operational mechanisms, with self-training relying on parameter updates and CoT guiding reasoning through input manipulation, we argue that both exhibit structural consistency in terms of information compression and entropy convergence. In CoT, while model parameters remain unchanged, the iterative optimization of the reasoning path set $R_t$ gradually leads to semantic-level distributional convergence over the reasoning space $\mathcal{R}$. As iterations progress, the proportion of reasoning paths containing correct subpaths $R'$ increases, and these paths tend to converge toward the correct answer cluster. This process results in a gradual increase in the posterior probability of the correct answer, reflecting a reduction in semantic entropy that parallels the entropy minimization objective in self-training.

Q4. How does "re-evaluate from another perspective" reduce reasoning similarity and entropy?

A4. The exploratory prompt encourages LLM to break away from the current local reasoning trajectory and actively explore subspaces within the generation space $\mathcal{R} \times \mathcal{A}$ that are semantically orthogonal to the original path. Such orthogonality is reinforced through key semantic constraints embedded within the prompt ("alternative perspectives"), prompting substantial strategic changes in the reasoning process. This process can further converge the answer space, thereby compressing the entropy of the overall candidate answer set.

Q5. Include stronger CoT baselines, harder datasets, and overhead comparison.

A5. We have newly incorporated two recent and stronger CoT methods, Active[1] and Nash[2], along with more challenging datasets, AMC2023 and Gaokao. Our method achieves higher accuracy while maintaining reasonable token consumption.

[1] Diao S,et al. Active Prompting with Chain-of-Thought for LLMs.
[2] Zhang Z,et al. Multi-Path Inference with Preference Equilibrium.

Q6. Adaptive iteration improves efficiency, but the underlying mechanism remains unclear.

A6. Adaptive iteration enhances efficiency via early stopping based on semantic entropy. When LLM shows high confidence, it dynamically halts to save computation. As shown in Table, our method averages 2.1 iterations (vs. fixed 3), with 55.9% of samples stopped early, reducing token while maintaining performance.