Response to reviewer BqFQ

We thank the reviewer for the insightful comments. We have addressed them with clarifications and experiments.

W1: multiple LLM calls per query; complex multi-turn reasoning tasks.

Response: We clarify that multiple query-based calibration is a common practice in the literature [31,37]. In our setting, since we don't have access to the internal states of LLMs, additional cost with multiple calls represents a necessary and reasonable trade-off to alleviate this information gap. For complex multi-turn reasoning tasks, our multi-query methodology can be viewed as a form of test-time scaling: investing more computational resources during inference to obtain better calibration and consequently more reliable reasoning traces.

W2: robustness of prompts

Response: We provide experiments to demonstrate the robustness of our method to prompt variations. We modified the expression of steering prompts to be more concise while preserving the core steering intent. For example, we simplified the "very_cautious" prompt as follows:

Original(SteerConf): Read the question, provide your answer and your confidence in this answer. Note: (1) The confidence indicates how likely you think your answer will be true. (2) You are making important decisions, thus you should avoid giving a wrong answer with high confidence. (3) You should be very cautious, and tend to give low confidence on almost all of the answers.

Modified(SteerConf_brief): Read the question, provide your answer and your confidence in this answer. Note: (1) The confidence indicates how likely you think your answer will be true. (2) Be very cautious, tend to give very low confidence on every answer.

We compare the modified and original versions of SteerConf with backbone LLM GPT-3.5 without CoT setting in the table below. Notably, SteerConf_brief achieves even lower ECE than SteerConf, and both outperform GPT-3.5-vanilla. These results demonstrate the robustness of our proposed method to prompt variations.

W3: The answer selection process in Sec 3.3 is not intuitive

Response: Our answer selection mechanism is motivated by the principle that answers with higher calibrated confidence should be selected when the model is more certain, and vice versa. Specifically, we compare the relative rank of confidence scores among candidate answers, as formalized in equation (6). Experimental results in Table 5 demonstrate that this selection strategy improves calibration. We will clarify this point in the revised version.

Q1: setting of the steering level

Response: To investigate the impact of steering levels $l$, we vary $l$ from $5$ to $3$ using LLaMA3-70b-4bit with CoT, and report results in the table below. "SteerConf-no_verys" excludes very_cautious and very_confident prompts, while "SteerConf-no_mild" excludes cautious and confident prompts. The results show that $l=5$ yields better performance than $l=3$. Increasing $l$ may further improve results but incurs higher query cost and requires designing more fine-grained prompts, which is non-trivial. In practice, we find $l=5$ offers a good balance. We will add this discussion in the revised version.

Q2: Clarify the argument in Line 186-187

Response: We clarify that existing study [37] focuses on answer consistency, not confidence score consistency. We argue that a model's internal certainty is reflected in both answer and confidence consistency. Therefore, our method is designed to consider both aspects. Empirical results in the experiments section validate this approach.

Q3: influence of LLM's instruction following ability

Response: Prior verbalized confidence works [31,37] did not address the impact of LLMs' instruction following ability. In our setting, the vanilla verbalized confidence method [31,37] already poses a challenge: if an LLM (e.g., small LLaMA-2 models) has poor instruction following ability, it may fail to produce answer-confidence pairs in the required format. Once an LLM can reliably perform verbalized confidence tasks, its instruction following ability is sufficient for our steering confidence instructions. Figure 4 demonstrates that our steering prompts are successfully followed by current LLM backbones. We will clarify this point in the revised version.

Q4: smaller models

Response: As suggested, we conduct experiments using the small model Qwen3-1.7b under the CoT setting, as shown in the table below. Due to the model's relatively weaker instruction following ability, frequent collapsed verbalized confidence elicitation renders results on some datasets (Law and Ethics) unusable; thus, we report results only on datasets with reliable responses. The results demonstrate that our proposed method consistently outperforms the vanilla verbalized confidence approach under the CoT setting, even with a small model backbone.

Dear Reviewer BqFQ,

Thank you for your effort in reviewing our paper. We have addressed your comments by conducting experiments on the robustness of prompts, varying different steering levels, and testing smaller model, all of which further demonstrate the effectiveness of our designs, and provided additional clarifications on the design and setting in our work accordingly.

Please have a look, and we look forward to your feedback. Thank you.

Best,
Authors of SteerConf

Response to reviewer NUu4

We thank the reviewer for the insightful comments. We have addressed all comments with experiments and clarifications.

W1: calibration curves

Response: We will add the calibration curves and move Figure 4 to the appendix as suggested by the reviewer. As external URLs to diagrams are not permitted in the rebuttal, these updates will be made in the revision.

W2: smaller and different model

Response: As suggested, we conduct experiments using the small model Qwen3-1.7b under the CoT setting, as shown in the table below. Due to its relatively weaker instruction following ability, frequent collapsed verbalized confidence elicitation renders results on some datasets (Law and Ethics) unusable; thus, we report results only on datasets with reliable responses. The results demonstrate that our proposed method consistently outperforms the vanilla verbalized confidence approach under the CoT setting, even with a small model backbone.

W3: compare this approach to more cautious prompts

Response: As suggested, we report results using only the very_cautious, cautious, confident, and very_confident prompts in the LLaMA3-70b-4bit with CoT setting, respectively. SteerConf consistently achieves the best overall performance in terms of average ECE and AUROC, though a single prompt could outperform on specific metric. We will include this comparison in the revised version as recommended by the reviewer.

Minor W4 & Minor W6:

Response: As suggested, we will rephrase the relevant sentence to improve presentation.

Minor W5: discuss a related work

Response: As suggested, we will discuss the suggested related work.

Q1: experiment with a greater/different number of steering levels

Response: As suggested, we vary the number of steering levels from $5$ to $3$ to assess their impact, using LLaMA3-70b-4bit with CoT. In the table, "SteerConf-no_verys" omits the very_cautious and very_confident prompts, while "SteerConf-no_mild" omits the cautious and confident prompts. The results show that $l=5$ yields better performance than $l=3$. In practice, $l=5$ offers a good balance, as increasing the number of levels incurs additional query cost. We will include this discussion in the revised version.

Q2: weighting the confidence and answer consistency terms

Response: We clarify that our approach multiplies the confidence and answer consistency terms, $c(x)=\mu_c\cdot \kappa_{ans}\cdot \kappa_{conf}$ as shown in Equation (4) below line 225. This multiplicative design avoids the need to manually balance the two terms, and leverages the transferability advantage of multiplication. Introducing a weight parameter would not directly assign a specific weight to either term.

Response to reviewer 1saN

We thank the reviewer for the insightful comments and we have addressed them as follows.

W1: semantic differences in the generated answers, more open or ambiguous questions

Response: We clarify that, following prior work, our current tasks focus on numerical answers (GSM8K) or answers with predefined options (Law), where ambiguity is minimal and answer consistency is both sufficient and effective. We agree that addressing semantic differences in generated answers is important for more open-ended tasks, and will include this discussion as future work in the revised version.

W2: Each prompt’s contribution
Q1: individual impact of each steering prompt

Response: As suggested, to assess the contribution of each steering prompt, we conduct an ablation study by removing each prompt individually using LLaMA3-70b-4bit with CoT as the backbone. For example, "SteerConf w/o very_confident" excludes the very_confident prompt. The results are shown in the table below. Notably, the cautious prompts (cautious, very_cautious) primarily improve AUROC, while the confident prompts (confident, very_confident) mainly enhance ECE.

Q2: Perform with CoT setting

Response: We highlight that Table 2 in our paper (Page 7, above line 260) presents results for the CoT setting across different model backbones. CoT prompting consistently enhances the performance of all LLMs. Importantly, SteerConf continues to outperform the vanilla verbalized confidence approach, confirming the effectiveness of our method regardless of CoT usage.

Q3: How does SteerConf perform when the prompts intentionally guide the model to express overly high or low confidence?

Response: The design principle of our method SteerConf is precisely to intentionally steer the confidence output of LLMs. SteerConf effectively handles cases when prompts intentionally guide the model to express overly high or low confidence. In fact, this intentional steering mechanism is the key design of our SteerConf (Steering Confidence) method as stated in Sections 1 and 3, where "steering" refers to: generating confidence scores in a specified direction (e.g., conservative or optimistic) by employing prompts with varying degrees of caution or confidence. Specifically, we apply steering prompts like "You should adopt a very cautious approach and assign low confidence to most of your answers" to guide the LLM to express deliberately calibrated confidence levels. The steered confidence scores and answers are then aggregated to obtain more reliable confidence estimates. We will emphasize this point in the revised version.

Response to reviewer ZMK7

We have addressed all your comments below with experiments. Thank you for the constructive comments.

W1: compare with baselines such as Avg-Conf and Pair-Rank

Response:

1. As noted in Line 267, the "Top-K" baseline already incorporates the Pair-Rank aggregation, which is specifically tailored for the Top-K approach.
2. For Avg-Conf, we report results using the LLaMA3-70b-4bit backbone with CoT in the table below. The results show that our proposed method outperforms all baselines, including Self-Random and Misleading variants with Consistency, Pair-Rank, and Avg-Conf aggregation. For instance, SteerConf achieves ECE of 7.5, notably better than Self-Random with Avg-Conf that is 14.3.

W2: make clear the effectiveness of each component. (1) steering prompts (2) different aggregation approaches

Response: We perform an ablation study to assess the impact of steering prompting and aggregation components, using LLaMA3-70b-4bit with CoT as the backbone. For (1), removing steering prompting yields "SteerConf w/o Steering Prompting". For (2), we replace our proposed aggregation with Consistency and Avg-Conf, denoted as "SteerConf+Consistency" and "SteerConf+Avg-Conf". Ablating both components results in the Self-Random approach with Consistency or Avg-Conf aggregation.
The results are reported in the table below, which demonstrates that each component contributes to improved performance, supporting the effectiveness of our method’s design.

Dear Reviewer ZMK7,

You have raised two points that are constructive and easy to address. We have addressed them with experiments. Please have a look, and we look forward to your feedback. We thank you for your effort in reviewing our paper.

1. Comparison with Pair-Rank and Avg-Conf. In the paper, we compared SteerConf to Top-K with Pair-Rank aggregation, and in this rebuttal, we have included additional results with Avg-Conf. Our SteerConf method outperforms these baselines, demonstrating its effectiveness.

2. Ablation Study. We have conducted ablation studies to show the effectiveness of each component in our method.

Best,
Authors of SteerConf